Halkalı merkez mahallesi fatih cd ozgur apt no 46 , Küçükçekmece , İstanbul , 34303 , TR
Create a customized robots.txt file for your website with Eka Server's free generator tool. The robots.txt file tells search engine crawlers which pages or files they can or cannot request from your site.
With our user-friendly interface, you can easily create a professional robots.txt file by selecting the options you need, without having to write the directives manually.
Control which parts of your site search engines can access
Keep sensitive content away from search engines
Guide crawlers to focus on important content
Include sitemap URL for better indexing
A robots.txt file is a text file webmasters create to instruct search engine robots (typically web crawlers) how to crawl and index pages on their website. It's part of the Robots Exclusion Protocol (REP), a standard used by websites to communicate with web crawlers and other web robots.
A properly configured robots.txt file can help you control which areas of your site are crawled, prevent search engines from accessing sensitive content, improve crawl efficiency, and generally improve your site's SEO performance.
No rules added yet. Use the form above to add rules.
Turkey (Türkçe)
Germany (German)
Worldwide (English)
# Standard robots.txt file
User-agent: *
Allow: /
Robots.txt is a directive, not a security measure. Bad bots can ignore these instructions, so don't use it to hide sensitive information.
The robots.txt file must be placed in the root directory of your website (e.g., www.example.com/robots.txt) to be effective.
Different search engines may interpret robots.txt files differently. Some respect all directives, while others only support basic commands.
Including your sitemap URL in the robots.txt file helps search engines discover and efficiently crawl your website's content.
Common questions and answers about robots.txt files
A robots.txt file should be a plain text file with directives in the following format:
User-agent: [name of robot]
Disallow: [URL path not to crawl]
Allow: [URL path to crawl]
You can have multiple User-agent sections to specify rules for different crawlers. The User-agent line specifies which crawler the rules apply to, with * being a wildcard for all crawlers.
Yes, but support varies by search engine. Google, Bing, and some others support the * wildcard (which matches any sequence of characters) and the $ character (which specifies matching the end of the URL). For example:
Disallow: /*.php$ (blocks access to all URLs that end with .php)
Disallow: /private* (blocks access to all URLs starting with /private)
Search engines typically check for an updated robots.txt file each time they visit your site, which can be daily for active sites. However, it can take anywhere from a few hours to a few weeks for all search engines to recognize and fully implement the changes, depending on how frequently they crawl your site.
Not necessarily. While robots.txt can prevent a page from being crawled, it doesn't guarantee that the page won't appear in search results. If other pages link to your blocked page with descriptive text, search engines might still index the URL without crawling its content. To completely prevent a page from appearing in search results, use the "noindex" meta tag or header.
Robots.txt controls whether a crawler can access a page, while meta robots tags control whether a crawler can index a page and how it should be presented in search results. A robots.txt file is a site-wide control at the server level, while meta robots tags are page-specific controls in the HTML code. For complete control, often both are used together in a coordinated SEO strategy.