I had this issue many times. Find the same occurrences of string between two different files so that you can pervent duplicates.
You should create foor loop that will take every line from file 1 and compare it to records in file 2.
for i in `file1.txt`; do grep -E "^$i\$" file2.txt; done
This will output only the exact strings that occurres in both files.