How to ensure request coming from browser? - Webmaster Forums - Webmaster forum for HTML, PHP, ASP, CSS and more

dhanasekaran1980 · 08-04-2006, 05:44 AM

Hi,

Can anybody help me how to ensure that the request is coming from the browser. The request many come from search engine or any java program. I need to ignore these requests.

So is there any way to find the request is coming from the browser or someother source through JAVA or JSP.

Thanks

Dhanasekaran.

ahk2chan · 08-12-2006, 06:43 PM

Well for the search engine piece you can actually define a robots.txt to disallow robots crawling those protected pages.

However, about browser... I think you can setup something in the httpd.conf (if you are using Apache) that will match the browser string provided by the "browsers". for example: "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)". Another not-as-safe way is to use javascript to check the browser type...

But at the end of the day, a browser is a "program" too... so users may choose to write a program / crawler that will send the same browser string like Firefox or IE...

rjplummer · 08-16-2006, 09:03 PM

It's pretty tough to come up with a regex that will match all valid user agants. Plus most search engines, at least in some circumstances, spoof browser user-agents to protect against click-fraud.

If you're comfortable requiring JavaScript, I'd use JavaScript to inject a value that tells you the client's running JavaScript which makes certain it's a browser, e.g.,

ASP/VBScript example:

<%
Randomize
If IsEmpty(Session("injectedvalue")) Then
Session("injectedvalue") = CStr(Rnd())
%>
<html><head><script language="JavaScript">
window.location = "?iv=" + escape("<%= Session("injectedvalue") %>")
</script></head><body>
Innocuous text for spiders, etc.
</body></html>
<%
ElseIf Session("injectedvalue") <> Request.QueryString("iv") Then
Response.Redirect "wherever you want to send non-browsers"
End If
%>
Protected page here

This needs to be a little more complex if the page already has a uri-query on it.

ahk2chan · 08-16-2006, 09:42 PM

To exclude all robots from crawling:

Code:

User-agent: *
Disallow: /

The javascript thing is a good and creative one! Thanks for the info!