How to fix "W: Duplicate sources.list entry"?

Your sources.list really has duplicate entries.

First Let's see the correct form of a repository line or source line:

The correct format of repository source line is

<type of repository>  <location>  <dist-name> <components> 

For example, a repo line can be like this one

  deb http://archive.ubuntu.com/ubuntu precise main

^{Here, it means, the repository is for binary packages, which are hosted in http://archive.ubuntu.com/ubuntu and this repository is for Ubuntu precise (12.04) and this repository contains the main (software which are officially supported by Canonical) component.}

• Type: The type can be deb and deb-src. deb means a binary repository where deb-src means a source repository

• Location: http://archive.ubuntu.com/ubuntu location of the repository.

• Dist-name: The distribution name of Ubuntu release. For Ubuntu 12.04 it is precise, for 11.10 it is oneiric.
  ^{You can visit the Ubuntu Wiki to view an updated list of Ubuntu releases and their code names.}

• Component: It can be main, universe, multiverse and restricted. These words indicates the level of supports for the packages and the licensing status.

See this page for more information.

Please take note that, you can add one or more component in a line, so "main", "universe", "restricted" and "multiverse" can be in a single line. Also note, Though you add more than one component in a single line, APT system considers them as separate line containing only one component.

So, If your sources.list have a line like this

deb http://archive.ubuntu.com/ubuntu precise universe

Then it can't have another line like the below (which your files have)

deb http://archive.ubuntu.com/ubuntu precise main universe

which is equivalent of these two lines

deb http://archive.ubuntu.com/ubuntu precise main
deb http://archive.ubuntu.com/ubuntu precise universe

Because, you are duplicating universe twice, so there will be an error for that duplicate. An error will be given for each duplicate found.

Solution:

After analyzing your sources.list file, I found that, It is a basic one which is supplied by Ubuntu by default. You can build a default file following the procedure:

1. Open a terminal and first rename the existing sources.list file to sources.list.bak file. (we can safely remove that, but caution is good).

    sudo mv /etc/apt/sources.list /etc/apt/sources.list.bak
   

2. Then open software-properties-gtk to build a new one. Type the below command in a terminal or in dash command prompt with pressing Alt+F2

   software-properties-gtk
   

   ^{Note, for Ubuntu version 11.10 or earlier you have to use gksu before the software-properties-gtk command}

3. After the window is open,

   • Select all four categories in Ubuntu Software tab
   • Select precise-security, precise-updates and precise-backports category in Updates tab.
   • Select Canonical partner and Independent category from Other software tab.

That's it. You have now a default sources.list file without error. Compare this file with the previous one, if you want.

Update to deal with sources.list.d dir's files

Sometimes a duplicate entry can be in a file in the /etc/apt/sources.list.d directory, which is also used by APT. So, you should look at that directory and see if there is any duplicate in those files. It is not necessary for duplicates being in the same file.

Example case:

A user had this error message showing while doing sudo apt-get update.

W: Duplicate sources.list entry http://dl.google.com/linux/chrome/deb/ stable/main amd64 Packages (/var/lib/apt/lists/dl.google.com_linux_chrome_deb_dists_stable_main_binary-amd64_Packages)
W: Duplicate sources.list entry http://dl.google.com/linux/chrome/deb/ stable/main i386 Packages (/var/lib/apt/lists/dl.google.com_linux_chrome_deb_dists_stable_main_binary-i386_Packages)
W: You may want to run apt-get update to correct these problems

But there was no entry with http://dl.google.com in the main /etc/apt/sources.list file. Looking at the /etc/apt/sources.list.d/ directory we found these files:

output of ls /etc/apt/sources.list.d/:

aims-sagemath-precise.list
aims-sagemath-precise.list.save
dropbox.list
dropbox.list.save
fossfreedom-packagefixes-precise.list
fossfreedom-packagefixes-precise.list.save
google-chrome.list
google-chrome.list.save
google.list
google.list.save
sagemath-monolithic-precise.list
sagemath-monolithic-precise.list.save

The files google-chrome.list and google.list were good candidates for containing duplicates. So, checking the contents of both files was necessary.

Output of cat /etc/apt/sources.list.d/google-chrome.list:

deb http://dl.google.com/linux/chrome/deb/ stable main  

and of cat /etc/apt/sources.list.d/google.list

deb http://dl.google.com/linux/chrome/deb/ stable main

So, there was a duplicate entries in the sources of Apt. Since google-chrome.list only contain a single apt line and it was also listed in google.list file we can safely remove this file with the command

sudo rm /etc/apt/sources.list.d/google-chrome.list

Then the problem was solved.

For Ubuntu 12.04

Graphically:

Press Alt + F2 and paste software-properties-gtk (or you can open "Software Center" then go to "Edit" > "Software Sources"). Go to tab "Other Software", choose the duplicate entry and press the "Remove" button.

If you need a command line option, here it is:

cat /etc/apt/sources.list | perl -ne '$H{$_}++ or print' > /tmp/sources.list && sudo mv /tmp/sources.list /etc/apt/sources.list  

The command does this:

cat reads the file and passes the content to perl which removes the duplicate lines. The result is then saved > in a temporary file which is then moved to replace the original /etc/apt/sources.list file.

How bad are duplicate entries in sources.list?

I don’t know how bad it is, but i don’t like sudo apt-get update showing me duplicate entries.

By the way its not that bad, its just showing you that you have duplicate entries.

SourcesList

The sources.list file is a key factor in adding or upgrading applications to your Ubuntu installation. This is also used by your system for system updates. The file is basically the roadmap for your system to know where it may download programs for installation or upgrade.

Its just like Windows update

You can remove duplicate entries in few easy steps with Y PPA Manager

 sudo add-apt-repository ppa:webupd8team/y-ppa-manager -y
 sudo apt-get update
 sudo apt-get install y-ppa-manager -y

Open y-ppa-manager form Dash

Enter You Admin Password

Double Click On Advanced

Scan & Remove Duplicate PPA's & Click Ok

It will take some time ( 1 or 2 Mints ) To Scan & Remove Duplicate PPA's

The partner repository is duplicated inside /etc/apt/sources.list and /etc/apt/sources.list.d/precise-partner.list.

Just remove those "precise-partner" files as the partner repository is already present in sources.list.

sudo rm /etc/apt/sources.list.d/precise-partner.*

Make a back up copy of your sources.list:

sudo cp /etc/apt/sources.list{,.backup}

Now remove the original and update:

sudo rm /etc/apt/sources.list && sudo apt-get update

I had the same problem, open software sources. unchecked "Canonical Partners". The Conflict is between "Canonical Partners" and " Canonical partners Added by software Center"

If you couldn't figure it out in your source.list, then do this:

1. Left-click on settings icon (at the extreme top corner of your screen) and select System Settings.
2. Click on Software Sources and move to Other Software.
3. Uncheck one of the 'Canonical Partners(Source Code) -software packaged by Canonical for their partners' files and one of the 'Canonical Partners -software packaged by Canonical for their partners' files also and click Close to leave.
4. Simultaneously press CTRL + ALT + T keys to open terminal and type sudo apt-get update.

10.10 and Earlier

Applications > Ubuntu Software Center > Edit > Software Sources > Other Software

OR

System > Administration > Synaptic Packet Manager > Settings > Repositories

Select and Remove duplicate entries.

For Ubuntu 9.10 - 12.04

There is an app called Y-PPA-Manager which can do that and much more.

You can install it like that from a Terminal:

sudo add-apt-repository ppa:webupd8team/y-ppa-manager
sudo apt-get upgrade
sudo apt-get install y-ppa-manager

After that, just start the application an go to the Advanced menu.

Then select "Scan and remove duplicate PPAs" and hit OK.

I searched for such a tool, but I didn't find any...
So, I ended up coding one myself with PHP.

Sorry for the dirty unorganized source code.

chkdup - Screenshot:

Author

Mubarak Alrashidi (DeaDSouL)

Requirements:

1. PHP.
2. sudo.

Installation:

1. Open the terminal.
2. To install PHP if you don't have it, type: sudo apt-get install php5 php5-cli.
3. Download the chkdup.
4. Extract the file anywhere you like.
5. Type: sudo sh install.sh.

Uninstall:

1. Open the terminal.
2. Go to the path you extracted the tool in.
3. Type: sudo sh uninstall.sh.

How to use it:

Just open the terminal and type sudo chkdup.

FAQs:

What does the install.sh do ?

• The installer will move the chkdup.php to /opt/ then will create a symbolic link in /usr/bin/.

What about the uninstall.sh ?

• The uninstaller will remove the following files:

  1. install.sh
  2. /opt/chkdup.php
  3. /usr/bin/chkdup
  4. itself uninstall.sh

Is it safe to use chkdup ?

• Yes it is, since it takes a backup of the current sources.list before attempting to do anything. But I'm not responsible if anything happened. The full responsibility will be yours.

License

GNU General Public License