2, it is assumed that the user visits a random page to start a collection of web pages, and then follow the link website website forward, not back to browse, then visit a page that has been the probability of the PageRank value of the website.

Factors that affect the value of PR
PR value is manifested in the 0-10 from the 11 values in the Google Toolbar by a horizontal green histogram shows that the case 0 white. It is not the site for the page, so a Web site’s home page and inside pages tend to have very different values of PR. Google market development process as a result of the reasons for the PR value of Chinese web pages than a whole page in English. Chinese web sites, has a basic 3 PR, 4 achieved PR count, 5 PR is good PR even if the 6,7-level Web site quite good. Of course, due to the impact of PR factors most directly from the link, this rating does not mean that the level of standard content, Web content quality of the impact of PR is an indirect, long-term. Algorithm based on the principle of PR value, we can see the impact of a web site (Home) PR value of the following main factors:

1, Quality into the site links
According to the analysis of high-quality incoming links, the PR value of a high need for a link from the following website:

Directory search engine and directory sites have joined the exchange of links

Obtained from the PR value of not less than 4 and with your theme related or complementary websites

Your link appears in the flow of large, well-known and frequently update the important site (such as news source)
Exchange links with your website link has little to export
High quality and content of the web link (PR values have increased the potential for large)

2, the number of export link
Calculated in accordance with the principle of PR, as a result of “the importance of a web page to be delivered to the average Web page it references”, so the other hand, a page of links will be exported too much potential value arising from the loss of PR on this page. However, from the content point of view, an appropriate number of themes associated with the Export link to the search engine a good impression. As a result, a page, especially the home page to grasp the number of export link, it should be taken into account the content of PR value and key words of the balance between the two that control the number of export links to appropriate not more than 10.

3, the search engine included a number of site pages
SEO explore, people place too much attention to the core words, as well as the performance of the home page, but often overlooked a very important question: namely, search engines for a number of sites included in the page. SEO which is also of critical importance. A query using the core words of the website ranked poorly, perhaps because a large number of pages to be crawled, use the other words in the user query, with the page was ranked the top three. As the user searches using the key word has spread, and make this situation very often to the site visits.

It is precisely because of the number of pages included a direct impact on traffic, so the impact on the PR value is also significant. The more pages were included, the higher the home page PR. But needs to be emphasized is that the number referred to here refers to the number and site were included the ratio of the total number of pages, rather than absolute Indexed pages. For example, a 50 page site is a collection of pages, the site were included the ratio is 1 / 5, a 5000 page site is a collection of 500 pages, the ratio is 1 / 10, with the result that included 10 pages Indexed than 500 pages of Web site rankings also more advantages. However, for various search engines, how to achieve the ratio of what is good, there is no relevant data.

Core Tip: This is a webmaster from the Chinese article, blog optimization, optimization of search than many of the great masters of the optimization speak out more authoritative manner, this is the real SEO recommendations, rather than cheating SEO.

To communicate with the network administrator often the case, a number of widely disseminated the idea may be correct in the past, but they might no longer be the current situation.  When we recently spoke with a few friends on the web site on the face of the structure of the situation.  One friend is worried that the use of dynamic URLs, or even that “the search engine can not deal with dynamic Web site.”  Another friend that dynamic Web site on search engines is totally not a problem, those are a thing of the past.  Another said that he never even do not understand the dynamic and static web site web site What is the difference between compared.  For us, this moment so we decided to look a good dynamic and static web site URL to this topic.  First of all, let us define what we need to discuss the theme:

What is a static URL?

A static Web site, as its name implies, is a web site will not change, it usually does not contain any URL parameters.  For example: http://www.example.com/archive/january.htm.  You can enter in the search box filetype: html in Google search for a static Web site.  Web site update this type of page would be more time-consuming, especially when growth in the amount of information very quickly, because a separate page for each must change the compiler code.  This is why site managers in dealing with large, frequently updated Web site, such as online shopping site, forum communities, blog or content management system, the reasons for the use of dynamic Web site.

What is Dynamic URL?

If the contents of a Web site stored in a database, and, upon request, to show the page, then you can use dynamic Web site.  In this case, the site content is provided by the template-based form.  Under normal circumstances, a dynamic Web site look like this: http://code.google.com/p/google-checkout-php-sample-code/issues/detail?id=31.  You can search for like? = & This symbol to identify dynamic Web site.  Dynamic web site has a defect is different from the same site can have content.  As a result, different users may contain links to the different parameters of the web site, but none of those sites contain the same content.  This is why network administrators sometimes want to be a dynamic URL to static URL re-written in one of the reasons why.

Should I make my dynamic URLs look static ones?

In dealing with dynamic Web site, we hope you understand the following facts:

1, it is necessary to correctly generate and maintain the Web site from the dynamic to static URL re-writing of the change is actually a very difficult thing.

2, the dynamics of the original Web site to provide us with more security, let us deal with issues such as detecting and avoiding problematic parameters of those things.

3, If you would like to rewrite the URL, please remove unnecessary parameters while maintaining its dynamic web site, please look.

4, If you would like to provide a static URL instead of dynamic URLs, then you should also generate the corresponding static content.

Static and dynamic Web site, Googlebot to identify which one better?

We come across many webmasters, like our friends, as that appears to be static or static Web site for the site is indexed and ranking advantage.  Such a view is based on the assumption that the search engines crawl and analysis contained in the session ID (session ID) and source tracking device (source tracker) when the site is open to question.  However, the fact is that Google has in these two areas of considerable progress.  On the click-through rate, the static Web site may be some slight advantage because users can easily read on the site.  However, the indexing and ranking, the use of database-driven websites does not imply a distinct disadvantage.  Compared to the parameters of hidden so that they appeared to be a static web site, we also hope that the dynamic web site will be provided directly to the site search engine.

Now, let us look at some of the dynamic web site to disseminate widely the views of, and to correct some fool hypothesis webmaster.  :)

Legend: “dynamic web site can not be crawled.”

The facts: We can crawl dynamic URLs and explain the different parameters.  If you order for the web site looks like static, and hidden to those who can provide valuable information on Google’s parameters, but doing so will give the site trouble crawling and ranking.  Our suggestion is: please do not change the format of a dynamic website so that it appears to be static.  As far as possible, use static URLs to display static content is desirable, but you decide to display in the case of dynamic content, please do not hide the parameters so that they appear static, as it would be removed which will in our analysis of useful information web site.

Legend: “The parameters of dynamic web site to less than 3 months.”

Fact: For the number of parameters and there was no limit.  However, a good experience is not to let your site is too long (this applies to all URLs, whether static or dynamic).  You can remove some of the important parameters for the Googlebot, look better to the user a dynamic Web site.  If you are not sure what parameters can be removed, we suggest that you will all of the dynamic URL parameters are provided to us, our system will find out which is not important.  Hidden parameters will affect our analysis of the right of your Web site, we can not identify these parameters, some important information may also missing.

Here we believe that some of you may doubt the existence of some of the problems

Here we believe that some of you may doubt the existence of some problems.

Does this mean that I should completely avoid rewriting dynamic URLs?

This is our recommendation, unless you can ensure that you only remove the superfluous parameters, or all those who may be able to have an adverse impact on the parameters of a complete deletion.  If you bring your own dynamic Web site looks like any modifications to static, you should make it clear that there is risk, there may lead to some information can not be normal to compile and identification.  If you want to increase your website version of a static, you must provide a true sense of the static content, such as those who are able to generate the corresponding path through the website and access to the document.  If you only modify the performance of dynamic Web site forms, and not to provide static content, then you have may be just the opposite.  Requests directly to the standard dynamic URL to provide to us, we will automatically identify the parameters of redundancy.

Can you give me you give an example?

If you have a standard format such as the following dynamic URL: foo? Key1 = value & key2 = value2, we recommend that you do not change it, Google will determine which parameters can be removed; or you can remove those unnecessary for the user parameter.  But to be careful to remove only those parameters is not important.  There are a number of parameters that contains examples of dynamic web site:

www.example.com/article/bin/answer.foo?language=en&answer=3&sid=98971298178906&query=URL

- Language = en – indicate the language of the article

- Answer = 3 – This article contains a number of 3

- Sid = 8971298178906 – the session identification code is 8971298178906

- Query = URL – so that the article was to find the query is [URL]

Not all of the parameters to provide additional information.  Therefore, the site will be rewritten to www.example.com / article / bin / answer.foo? Language = en & answer = 3 may not cause any problems, as all irrelevant parameters are removed.

Here are some after that seems to be a modification example of a static Web site.  Compared to not rewrite, direct provision of dynamic web site, these URLs may cause more crawling problems.

www.example.com/article/bin/answer.foo/en/3/98971298178906/URL

www.example.com/article/bin/answer.foo/language=en/answer=3/ sid = 98971298178906/query = URL

www.example.com/article/bin/answer.foo/language/en/answer/3/sid/ 98971298178906/query/URL

www.example.com / article / bin / answer.foo / en, 3,98971298178906, URL

If you re-written in a dynamic web site example, as noted above, he will probably lead us to crawl a lot of unnecessary, because these URLs contain session ID (sid) and query (query) parameters of the variable value, which the invisible generated a lot of different looks of the URL, and their contents are the same.  These formats let us through this web site is difficult to understand the actual content of the return URL and parameters and is unrelated to 98971298178906.  However, the following examples of this rewrite, however, has nothing to do all the parameters are removed:

www.example.com/article/bin/answer.foo/en/3

While we can accurately deal with this site, we encourage you to use or not to rewrite this.  Because it is hard to defend, but as soon as a new parameter was added to the original dynamic URL, then the site will need to update immediately.  Failure to do so, then it will once again lead to the generation of the parameters of a hidden web site looks like a static URL.  So the best solution is usually to maintain a dynamic web site to their original appearance.  Or, if you remove irrelevant parameters, remember to keep this web site is dynamic:

www.example.com/article/bin/answer.foo?language=en&answer=3

We hope this article to you and our friends there to help, so that dynamic web sites around the definition of speculation.  If you have additional questions, welcome to join in support of our.

Here is a quote from that post that summarizes the change:

So what happens when you have a page with “ten PageRank points” and ten outgoing links, and five of those links are nofollowed? Originally, the five links without nofollow would have flowed two points of PageRank each (in essence, the nofollowed links didn’t count toward the denominator when dividing PageRank by the outdegree of the page). More than a year ago, Google changed how the PageRank flows so that the five links without nofollow would flow one point of PageRank each.

In other words, the usage of the nofollow attribute will still stop the page that is being linked to from receiving PageRank and anchor text value, but the PageRank that is being “saved” with the nofollow links won’t flow to the other links present in the linking page.

According to Matt Cutts, the change was implemented to discourage website owners who were using a technique called PageRank sculpting, which is the process of using the nofollow attribute internally to change the PageRank flow among different pages. The objective is to give more PageRank value to some pages (the ones that sell something, for example) at the expense of less important pages (the Privacy Policy or Terms of Service, for example).

If Google wants to discourage PageRank sculpting, I am fine with that. The problem that I see with the change is that it might affect other types of websites negatively, blogs above all.

Why?

Because now all the links on your comment section are reducing the PageRank that would otherwise flow to your internal pages.

Here is a numerical example to let you understand it. Suppose that you write a super cool post explaining “Why Obama really won the U.S. presidential election.” The post gets some exposure on social media, and it ends up receiving 200 backlinks from other sites and blogs. Let’s assume that, on average, each backlink was carrying 5 points of PageRank, so the total points of PageRank for that post would be 1,000.

Now let’s suppose that you have 20 navigational links on your layout (e.g., the main menu, the categories and a section with popular posts). This means that each of those internal links would carry 50 points of PageRank to other internal pages on your site, which is good.

The problem comes when people start commenting on the post. Let’s say that, given the popularity of the article, it ends up with 150 comments. Each of those comments carries a link to the site of the comment author. Despite being nofollowed, they would still be counted on the PageRank distribution of that page. Now we have 170 outgoing links, so each link receives only 5.9 points of PageRank. Each of your internal links will only pass 5.9 points of PageRank now, and the total points of PageRank that will remain inside your website are 118.

In other words, before Google changed how it handles the nofollow attribute you would have gained 1,000 points of PageRank with that post. With the change, this number is reduced to 118.

And that is just what would happen in one post. What if we consider a blog with 1,000 posts and tens of thousands of comments? Such a blog could be losing a huge amount of PageRank.

Google might do something to fix this problem, but we still got to hear about it.

Is there any solution? Yes, you can either close your comments completely (but I wouldn’t recommend that), or use an iframe or Javascript to host your comments, so that the links to the comment authors won’t be inside your pages. Andy Beard wrote a post on the topic titled Disqus – Why 95% Of Bloggers Should Switch.

If changing to an iframe or Javascript based solution could give a huge PageRank boost to my blog, I will seriously consider it. For the moment, though, I will wait a couple of weeks to see how the debate around the nofollow changes will evolve. I will keep you guys updated if anything new emerges.

First, Web content to increase in the key words.
Keywords 1.URL appear (in English, the Chinese also have a role in Pinyin)

2. Page title, keywords tag, description tag appeared Keywords

Keyword here appears to be natural, not keyword stuffing.

3. The content of naturally occurring Keywords

Can be very natural to add some keywords up.

4. The first and the last paragraph of the content appeared Keywords

Page top and bottom of the page are some good keywords, you can look at this website.
5.H1, H2 tags appear Keywords

6. Export link anchor text contains keywords

7. Picture of the file name that contains keywords, ALT attributes arise Keywords

ALT attributes to good use, but also conducive to the top.

8. Key words density 2-8% (this is different)

I think that about 3% better in general. Of course there are different requirements for different stations.

9. Of words bold or italic

10. With a comment function, comments appeared

Core Tip: repetitive content filtering search engine to search for users in order to provide a better search experience, if we search a keyword, the search results in a large number of different sites of the same content, I think this is not the search engine would be welcome.

Engaged in search engine optimization of the staff are very clear and original content for the significance of search engine optimization, especially in the second half of 2008, Baidu for duplicate content pages more stringent filtering, and search engines filter the content is repetitive in order to search provide users with a better search experience, if we search a keyword, the search results in a large number of different sites of the same content, I think this is the search engine will not be welcome. Search engine duplicate content on the originality of the content filtering and identification, mainly through the Hyperlink fingerprint analysis and information technology, to determine whether an article belongs to original, original content for the search and whether the user has what kind of significance?

Original content for the search engines mean?
More trust in search engines and like to include a large number of original content sites, because they hope that through these sites to get more content, rather than repeat the contents of a large number of early, many people use the procedure to collect the content of other sites, to enrich its own Web site content to advertising through the flow area of the goal, so there is duplication of content on the Baidu increasingly stringent review, if a large number of duplicate content sites, may result in the search refused to record the contents of this web site or search engine did not position that to go unpunished, since the content for these users is meaningless, but a large number of original content for the site, search engines will give a higher weight, and record the contents of these pages for better rankings. Therefore, a lot of regulators trying to reprint the content of other web sites, and then proceed to the second editor, is the so-called pseudo-original, which seems to have some effect, two years ago, in this way is very effective, but now we may not be Nevertheless, search engine technology teams are constantly study how to better filtering of duplicate content, to provide users with a better search experience, to some extent, the search engine’s duplicate content filter technology is more mature, and of course, later will be even better. Therefore, high-quality original content for SEO is very important.

Original meaning of the content for visitors?
We are always considering how to improve the website’s ranking, consider how to cater to search engines like, if one day we do not rely on search engines get traffic, your site will be some people like it? Our Web site content provided by search engine not only to meet the like, and the more content we offer visitors whether you really need them, it really meets their needs. If we provided the content is only to deceive the search engines, there is no way to meet the needs of users, we have lost the meaning of building site eventually flow needs conversion, so if it is simply the volume and position, will be meaningless. Therefore, content quality is not just original enough, more users really need high-quality content.
PIC

SEO project implementation – can not make bricks without straw
SEO project implementation with the entire team, every detail is important, high quality SEO content is in the process of project implementation an important part of the actual process of implementation of SEO projects, many teams have taken his words to the editor in accordance with save to write the content, but for a small team, this is unrealistic, and then the staff in terms of search engine optimization, high-quality content is very important, so we in the SEO process of project implementation, it is necessary to consider how to make more valuable premium content, the following is my personal project implementation on the SEO process, a major factor in the proportion of plans:
PIC
Excellent search engine marketers is to focus on the needs of the market by means of Internet marketing SEO, and not only for the traffic and ranking, if it is only for the traffic and ranking, I am SEO strategy that can be resolved through the search engine filters repeatability content, and then it is unfriendly to visitors, so our ultimate goal is through the web site marketing and promotion, rather than simply the number of order flow.

What is Google Page Rank and how does it affect me you ask. Google Page Rank is a website score that represents how important a websites relevance is in the category of business they represent. Google gets this score from measuring the amount of quality links pointing to your site from other websites with the same relevance.
If you have a link to your website from another website with a Google Page Rank of 5 and is relevant to the subject matter your company represents, then Google sees that as a quality link. If you can get your link on a homepage of a high ranking website that is relevant to yours, then you will get a higher score than putting it on a sub page of the site which will usually get a lower Page Rank.
Google usually makes a major Page Rank update every three to four months. Its servers around the globe have to compare there information and the Page Rank is not complete until all of Googles servers have been updated with the new Page Rank scores.
The Page Rank is measured from 1 to ten one being the lowest score and ten being the highest you can have. New sites and sites that have not had any search engine optimization applied to them will have a score of 0. If you use Mozilla Firefox for browsing you can install a new add called SEO Quake. This tool will allow you to see how many links you have in Google and Yahoo.
It will show you the sites page rank and when the site was registered. You can do a search in Google for new music for example. The free results will show up with all the link info and page rank score below the text listing. You can compare all the sites for yourself and can make a partial determination that the Google Page Rank is still a large factor is determining your placement for targeted keywords in the free listings.
I currently manage the SEO campaigns for 27 successful websites and have noticed my Google placement on all staying consistent in their positions as Yahoo and MSN positions continue to rise. Google has made several updates on my back link quality and count on each site but I see little movement in the sites placement since April when the last PR update was performed. The page rank is more than just a number and if it is that irrelevant than Google has not made many other major updates on its server since April either.
Just remember, optimize your site for your customers first and for the search engines second. You can get plenty of traffic using the wrong keywords but visitors will see your site is not what they were looking for when they get there. A few quality back links are more relevant than a bunch of non relevant back links. Use common sense and consider what your customers are looking for and what they expect to see when they get to your website.

Home
Web site’s home page (home or index page, etc.) should be the form of text, rather than flash and so on. Inside this text to include your target keywords or target phrase.

Tags
This is the header title tags, which should include you in this most important goal of keywords.

Meta Tags
‘description tag’ (label description) and ‘keyword tag’ (Keywords tab) should contain your target keywords or target phrase, but do not repeat.

Alt Tags
Important site picture, including: logo, photo, picture, such as navigation, it is necessary to add a simple description.

Key words text
In your site, there should be some contains high keyword density pages. However, do not use this approach too far, or else they would be cheating as a search engine and counter-productive.

Submitted to the search engine
Please do not use automatic submission software or services. In fact, there is no need to hundreds of search engines to submit your web site. The five major search engines and directories occupied 90% of search traffic. Familiar with the warmest regards to the number of manual search engine submission of your Web site is enough, it can not you spend a few minutes.

Links
And peer-Link site, content, PR value than their own site the best high. Do not join the link farm (Link Form), as was the search engine will let you blacklisted

I have been using dawjee.com to analyse search terms for a while now – toying with keyword suggestions and looking for interesting patterns or anomalies in the result data (e.g. pages with query string data often report an erroneous Google Page Rank of zero).

After discussing some findings with a colleague, I realised a lot of what I’m learning will also be of use and/or interest to other webmasters. I shall therefore catalogue all future findings in a series of articles / blog posts.

This first article shall observe the current Google results for the term “dandruff”.

At the of writing, the top 3 results are:

Position 1

Title : Dandruff – Wikipedia, the free encyclopedia

URL : http : // en . wikipedia.org/wiki/Dandruff

PR 6

Position 2

Title: Dandruff – MayoClinic.com

URL: http : // www . mayoclinic.com/health/dandruff/DS00456

PR 4

Position 3

Title: Dandruff

URL: http : // www . coolnurse.com/dandruff.htm

PR 5

(PR = Google Page Rank)

What’s interesting is that P2 (Position 2) has a lower PR (Google Page Rank) than P3 (Position 3). Both web pages contain Dandruff in their TITLE tags and urls. So a cursory glance would suggest that P3 should have a better (i.e. lower) rank than P2.

From taking a look at both pages, both are on topic and contain genuine information about dandruff. P2 is only a small page, and is part of a full article (i.e. many small interlinked dandruff pages). While P3 is a full article on a single page.

The keyword densities are 4.86% for P2 and 6.39% for P3. Neither page makes obvious use of contrived keyword stuffing.

However, if we drill down into the data for each result (use the small magnifying glass next to each result, or follow the link the Resources list below), we’ll find some significant differences in their hosts and backlinks.

P2 has 6 Google backlinks, with 4,990 to its host. While P3 has 3 and 270 respectively.

P2 also dwarfs P3 in terms of the number of pages Google has indexed (Pages in Host), and also has a higher host page rank 7 (vs 5 for P3).

To summarise, P2 may have a lower page rank, but it has a few more links and its home page is overwhelmingly more popular than P3’s. It is also part of a set of pages about dandruff. Instead of having the full article on a single page, it has been split among many. P2 therefore has several interlinked pages about dandruff.

It therefore appears that the pages hosted on popular (in terms of search engines) web sites can rank better than other pages, even if they have lower Google Page ranks. Splitting up articles into many small pages helps as well.

You can on your site to create a plain text file robots.txt, statements in this document in the site visit do not want to be part robot, so that the site of some or all of the content on the search engines do not have to be included, or designated search engine only the contents of the specified record.  robots.txt file should be placed on the root site.

When a search robot (some call search spiders) to visit a site, it will first check the site under the root directory of the existence of robots.txt, if it exists, the search robot will be in accordance with the contents of the document to determine the visit scope; If the file does not exist, then the search robots to crawl along the link.

robots.txt file:

“Robots.txt” file that contains one or more of the records through the blank lines to separate (by CR, CR / NL, or NL as at the end), each record format is as follows:

“<field>: <optionalspace> <value> <optionalspace>”.

In that paper, you can use # for comments, the specific methods and the use of UNIX in the same practice.  The document is usually recorded in one or more lines of User-agent year, followed by a number of Disallow lines, as follows:

User-agent:

The value of the search engine robot is used to describe the name of the “robots.txt” file, if the number of User-agent records that have more than one robot will be subject to the restrictions of the agreement, the document, at least There is a User-agent record.  If the value is set to *, the agreements are valid for any robot in the “robots.txt” file, “User-agent: *” This can only have a record.

Disallow:

The values do not wish to be used to describe a visit to the URL, the URL can be a complete path, it could be a part of, any Disallow the URL at the beginning will not be access to the robot.  For example, “Disallow: / help” on / help.html and / help / index.html search engines are not allowed to visit, and the “Disallow: / help /” allows robot to visit / help.html, and not be able to access / help / index . html.  Disallow any record is empty, that all parts of the site are allowed to be visited, in the “/ robots.txt” file, at least record a Disallow.  If the “/ robots.txt” is an empty file, then for all the search engine robot, the site is open.

For example the use of robots.txt file:

Example 1. A ban on all search engines to visit any part of the site to download the robots.txt file User-agent: * Disallow: /

Example 2. To allow the robot to visit all (or also can be used to build an empty file “/ robots.txt” file) User-agent: * Disallow:

Example 3. To prohibit access to a search engine User-agent: BadBotDisallow: /

Example 4. Permit to visit a search engine User-agent: baiduspiderDisallow: User-agent: * Disallow: /

Example 5. A simple example in this case, there are three directory of the Web site search engine to limit access so that search engines will not visit the three directories.  It should be noted that a directory for each statement should be kept separate, and not written in “Disallow: / cgi-bin / / tmp /”.  User-agent: * after a special meaning, representing “any robot”, so the document can not be “Disallow: / tmp / *” or “Disallow: *. gif” it was recorded there.  User-agent: * Disallow: / cgi-bin/Disallow: / tmp / Disallow: / ~ joe /

Robot special parameters:

1. Google

Allow Googlebot:

If you want to block in addition to all the roaming outside Googlebot to access your pages, you can use the following syntax:

User-agent: Disallow: /

User-agent: Googlebot

Disallow:

Googlebot follows the line of its own point, rather than point to the line of all robots.

“Allow” extension:

Googlebot identifiable as “Allow” extension of the robots.txt standard.  Other search engine bots may not be able to identify this extension, so please use your interesting to find other search engines.  ”Allow” the role of line with the principle of “Disallow” line, like.  You want to allow only listed in the directory or page you.

You can also use “Disallow” and the “Allow”.  For example, to intercept a subdirectory in the page other than all the pages, you can use the following entries:

User-Agent: Googlebot

Disallow: / folder1 /

Allow: / folder1/myfile.html

These entries will be in addition to intercept folder1 directory of all the pages outside myfile.html.

If you want to block Google’s Googlebot and allow the other robots (such as Googlebot-Mobile), can use the “Allow” rules to allow access to the robots.  For example:

User-agent: Googlebot

Disallow: /

User-agent: Googlebot-Mobile

Allow:

Use * to match its character sequence:

You can use an asterisk (*) to match the character sequence.  For example, to block all private visit at the beginning of the subdirectory, use the following entries:

User-Agent: Googlebot

Disallow: / private * /

To block all contain a question mark (?) Visit the web site, you can use the following entries:

User-agent: *

Disallow: / *?  *

Using the $ character matches the end of the URL

You can use the $ character the end of the URL specified with the matching characters.  For example, to block to. Asp at the end of the URL, you can use the following entries:

User-Agent: Googlebot

Disallow: / *. asp $

You can match this model used in conjunction with the Allow directive.  For example, if?  That a session ID, you can exclude all of the URL contains the ID to ensure that Googlebot will not crawl duplicate pages.  However, in order to?  At the end of the URL may be that you want to include the version of the page.  In this case, the robots.txt file can be set as follows:

User-agent: *

Allow: / *?  $

Disallow: / *?

Disallow: / *?  And his party will block contains?  Website (specifically, it will block all your domain name at the beginning, followed by any string, followed by a question mark (?), And then the string is arbitrary URL).

Allow: / *?  $ And his party will be allowed to contain any?  At the end of the web site (specifically, it would allow to include all your domain name at the beginning, followed by any string, followed by a question mark (?), There is no question mark after the character of the site).

Sitemap Site Map:

Site Map for the support of the new approach is the robots.txt file, including direct links sitemap file.

Like this:

Sitemap: http://www.eastsem.com/sitemap.xml

Expressed support for the current search engine company Google, Yahoo, Ask and MSN.

However, I would suggest or submit to Google Sitemap, which features a lot of links you can analyze the state of

Robots.txt benefits:

1. Almost all the search engines gives Spider follow robots.txt crawl rules, search engine Spider agreement to enter a Web site that is the entrance to the site’s robots.txt, of course, the prerequisite is the existence of the website this document.  Robots.txt is not configured for the site, Spider will be redirected to a 404 error page, the relevant studies have shown that if the site uses a custom 404 error page, then the Spider will be regarded as its robots.txt– although the is not a pure text file – Spider Index This site will bring big problems, the impact of search engine included on the site page.

2. Robots.txt to stop the unnecessary occupation of the search engines valuable server bandwidth, such as email retrievers, the majority of this type of search engine sites is meaningless; Another example image strippers, for most types of non-graphics Web site for its and has little significance, but a considerable amount of bandwidth.

3. Robots.txt to stop search engine to non-public page crawling and indexing, such as the site background processes, management procedures, in fact, for some in the operation of the site have a temporary page, if not configured robots.txt , search engines and even those temporary files will be indexed.

4. For the rich, there are many pages of web sites, configure the robots.txt is more important significance, because very often a search engine of its Spider face tremendous pressure to give Web site: Spider-like visit to the flood, if not checked and even affect the normal web site visit.

5. Similarly, if the existence of duplicate content sites, use the robots.txt page limit will not be part of search engine indexing and recorded, can be avoided by the search engine site duplicate content on penalties to ensure that Web site’s ranking will not be affected.

the risks associated with robots.txt and solutions:

1.?????everything, robots.txt at the same time also brought a certain degree of risk: the attacker also pointed out the site’s directory structure and location of private data.  Although the Web server’s security configuration properly under the premise of this is not a serious problem, but those ill reduced the difficulty of the attack.

For example, if the site privacy data www.yourdomain.com / private / index.html visit, then the settings in the robots.txt may be as follows:

User-agent: *

Disallow: / private /

In this way, an attacker can simply look at robots.txt to know the content you want to hide where the input in the browser will be able to visit our www.yourdomain.com/private/ did not like the content.  Of this situation, the general approach taken is as follows:

Set access permissions on the / private / content in password-protected so that attackers will not be able to enter.

Another approach is to the default directory changed its name to the main document index.html other, for example, abc-protect.html, so that the content will become the address www.yourdomain.com / private / abc-protect.htm, At the same time, the production of a new index.html file, the content along the lines of “you do not have permission to access this page” like, so that an attacker because I do not know the actual file name and do not have access to private content.

2. If the settings wrong, will lead the search engine will index all the data deleted.

User-agent: *

Disallow: /

The above code will be banned from all of the search engine index data.

Currently, the vast majority of search engine robots have to comply with the rules of robots.txt, and the Robots META tags are not currently supported, but is gradually increased, such as the well-known search engine on the full support of GOOGLE and GOOGLE also adds a command “archive”, can be restricted to whether or not to retain GOOGLE snapshot page.  For example:

<META NAME=”googlebot” CONTENT=”index,follow,noarchive”>

That crawl the site page and link pages to crawl along, but not to keep GOOLGE web page snapshot of the page

ShareThis

Source
How to use Robot.txt

Source
How to Increase Page Rank: Another Google Myth Revealed