Apache, Accessing files as a directory, Bug?

I found a problem with my Apache webserver:

Suppose I ONLY HAVE ONE file on my website (index.php):

example.com/index.php

When I access the URL using this way:

example.com/index.php/abc or example.com/index.php/123

Apache will display the content of: example.com/index.php

It has created a lot of problem to my web application. How can I make this as a 404 error?

Notice that my Apache is standard/default. I didn't do any modification to my Apache settings. OS is Fedora Linux.

Thanks.