merge 2 files line by line and delete duplicate words - shell script , awk

The following shell script main aim is to merge two files line by line and delete the repeated word or duplicate word from each line.

file 1 :
apple goa orange
hello hi how are you
file 2 :
mango goa orange apple
how are you hi hello
Final out put required is :
apple goa orange mango

hello hi how are you
Screen outputs :
nc041031:/opt/krish # cat 1.txt
apple goa orange
hello hi how are you
nc041031:/opt/krish # cat 2.txt
mango goa orange apple
how are you hi hello
nc041031:/opt/krish # cat merge.sh
#!/bin/sh

awk 'NR==FNR{a[FNR]=$0;next} {print a[FNR],$0}' 1.txt 2.txt | tee a.txt
awk '{ while(++i<=NF) printf (!a[$i]++) ? $i FS : ""; i=split("",a); print "" }' a.txt | tee final.txt

nc041031:/opt/krish # ./merge.sh
apple goa orange mango goa orange apple
hello hi how are you  how are you hi hello
apple goa orange mango
hello hi how are you
nc041031:/opt/krish # cat final.txt
apple goa orange mango
hello hi how are you


2 comments to "merge 2 files line by line and delete duplicate words - shell script , awk"

Post a Comment

Who ever writes Inappropriate/Vulgar comments to context, generally want to be anonymous …So I hope U r not the one like that?
For lazy logs u can at least use Name/URL option which don’t even require any sign-in, good thing is that it can accept your lovely nick name also and URL is not mandatory too.
Thanks for your patience
~Krishna(I love "Transparency")
వీలయితే నాల్గు పోస్టులు ...కుదురితే ఒక కామెంటూ ...

Translate

Enter your email address:

Buffs ...

Visitors


hits counter
View My StatsCheck Google Page Rank

Add to Google Reader or Homepage

Bookmark and Share

Tags


Powered by WidgetsForFree