In other words, if you write a testimonial for a product you have never used, you might get busted. This part is the one that makes sense. The controversial one is the “failure to disclose any conflict of interest” one. This could involve bloggers who earned a freebie from a company and ended up writing about it (without disclosing the freebie), for example. Furthermore, it could also mean that you can’t use any affiliate links inside your blog or website without disclosing that you stand to earn money if someone clicks on them.

I believe that trying to enforce strict guidelines and clean the web from spammers and scammers is a good thing. However, I am not sure if this proposed approach is the right one. It kind reminds me of the RIIA, and how they were trying to stop illegal music downloads by suing the heck out of a women who had shared a couple of songs via P2P….

Aaron Wall wrote a really good post about this topic, titled FTC Going After Bloggers = Epic Fail. Here is a quote from it:

What is absurd (to me at least) is how inefficient this process is. What needs to happen is better enforcement on ad networks, search engines, and merchants. Follow the money downstream rather than hunting for nickels upstream.

The people who are making fake sites are doing so because they are paid to. And amoral ad networks that syndicate ads based on *maximizing yield efficiency* (like Google AdWords) are designed to syndicate fraud because it is easy for advertisers to pay a lot for ads when their profit margins are nearly 100% because they scam people.

I completely agree with him. Going after individual bloggers will not solve the problem. It will just make the money shift hands, but the system will remain intact. Either way it is important to keep an eye on the development of this issue, because it might affect all of us.

The two main ingredients to rank highly in the search engines are that you must have good content and have lots of links pointing to you. For the vast majority of bloggers, writing outstanding content is not the problem. The problem is finding the best ways to build links to the blog, as well as building deeplinks to the blog posts themselves.

Thousands of self-proclaimed “link building experts” will try to sell you costly ebooks showing you the greatest places to build your links. The main reason this article was written is to show you that there is no reason to buy your way to a great link building campaign when you can do it all totally free. As a prologue, you should know that the most common free link building strategies such as link exchanges or linkfarms can be dangerous to your sites search engine rankings. For more details go to www.your-own-blog.com. As a good way to judge whether or not a link building technique will be dangerous to your site or not is to imagine if that link would be placed there if the search engines didn’t exist. If you were not adding your link to a site for the sole purpose that Google would find it, would you still add it?

With the prelude out of the way, lets move on. Building links to your blog is a lot different than building links to a static site. According to most people who own blogs, it is hundred of times easier. On your blog, you can build tons of links to hundreds of different pages, which you cannot do on a static site. Blogs are also updated on a daily basis, meaning the search engines crawl through them for hours and rank them higher with very little extra work. Your link building campaigns will be far more reaching than if you were on a static site. Every single post you make on your blog has a chance to gain tons of social media links which will result in higher search engine placement and more traffic. The more traffic, the more money you make!

After spending many hours searching the internet for places to build links for blogs, the best place you should focus your blog’s link building campaigns are on social media sites. These sites generally have a huge amount of visitors and the content that is promoted ranks heavily on the search engines. You are allowed to submit every blog post you write to every single social media site, thus building many links for every single blog post you write. This will allow you to have many deeplinks which will raise the authority of your blogs main domain. For more information login to www.blog-and-png.com. In the end, you will be able to rank higher for your blog posts without doing any extra marketing. Stop paying for bad quality ebooks when with a few minutes every day you can submit your blog posts to social media websites for free and receive far more benefit. If the social media sites like your blog posts, it may even make their front page resulting in thousands more visitors and hundreds more links. The result of this can easily snowball into reaching wide new audiences and rank for longtail keywords that you never thought possible. Give it a try because after all, you have nothing to lose.

Effective advertising that is tailored to reach target audience is necessary for the growth of business. The brand attains response and position, which can combat the growing competition. The competition arises due to the similar products and services floating in the market. The quality may or may not be the same but it surely affects the brand positioning. For achieving a prevailing position in the market, services of an online advertising agency has to be utilized.

The online advertising agency is like a channel partner for advertisement and promotions. The advertisers can have the advantage of reaching relevant target audience within a lesser ad spend. This is possible with the interactive ad solutions provided by an online advertisement agency. When it comes to the benefits, extensive coverage and relevant responses are few of them. Now, targeting the right audience is somewhat an interesting aspect. The solutions like Capping and Geo-Targeting are used by agencies. It helps in tracking the right market and group of users.

An online advertising agency also makes efforts to place the ads smartly. The planning related to size of the banner ad and its placement is necessary to attain good user response. The agencies obtain the traffic status of the publishers and theme of their website. In addition to it, the publisher website is checked for the quality and quantity of content. The keywords used by the publisher websites are also a matter of concern to the agencies. The more relevant the keywords will be, more relevant will be the traffic. It will help in coming across relevant clicks and impressions.

The clicks made by users are paid by the advertiser to the publisher, according top the laid conditions. Getting relevant traffic as audience ensures good return on ad spend (ROAS). This enables the advertisers to get canalized traffic and business leads. Most advertisers acquire leads and then approach the prospective customer directly. The interactive advertising provided by online advertisement agency increases brand awareness.

The online advertising agency associated with extensive banner network is useful for business organizations. Such an agency ensures a better positioning of brands. In this way, costs incurred by advertisers remain low. The advertisers enjoy good revenue against this expenditure.

Google was always first to introduce cool and great tools to Internet surfers. Gmail is the best example of it. I trust that Gmail is one of the most trusted and reliable email service in the world. Apart from this, Google have created tons of applications for all of us and that’s free of cost.

Just Look at some Google Tools and Applications

· Google Docs

· Google Mail (Gmail)

· Google World

· Orkut

· Google Alerts

· Blog Search

· Google Adword

· Google Adsense

· And more and more and more….

And I know that you know more services may be more then me.

With all the powerful and extra ordinary existing tools and applications created, Google is going to add a new Family Member to their Tools & Application List and that is Google Wave. Yes Google Wave is new web application that is developed by the same team who created the most used maps application in the world “Google Maps”

This time this team gets assigned to create such a magical application that will change the meaning of communication on the Internet. There are a lot of IM tools on the web including Gtalk from google then why Google needs to add Google Wave?

Google Wave is different. You even would not get to know that you are on a browser. Yes Google wave will run on a browser any browser like IE, Firefox or Chrome.

While I am finishing this post Google Wave is still in Development & it’s yet not ready for General Users.

The technique you commune via Google Wave will be lightening fast and will be real time. You will feel that you are using something like a future tech gadget lolz J. Well you will get to know why I said future tech gadget by the end of this year when Google Wave will be available for General Users. Google wave is independent from any frameworks and can be run on any browser as I said earlier in this post. You don’t have to install any thing to make run the Google Wave.

Well I know my text would not let you see what I am all talking about so I added the developer’s preview video at the end of this post. See that video and check the demo of Google Wave extra ordinary features and functions that will take you in the next stage of communication.

There are lot of fresh features that are added in Google Wave which were not available yet in any other application as far my knowledge. Let me give you some tips on that.

1. Advance Spell Checker

2. Conversation Tree structure

3. Conversation History

4. Collaborative Document Editing

5. Drag & Drop Uploader

6. API Development

You can embed the Google Wave in to any site or blog so that you can interact with your site audience from one platform. You can add more workflows like MySpace, My favorite Twitter or Facebook etc… using the API service in Google Wave.

If you are a developer and want to get into Google Wave before its launched then go to Google Wave API developer Account Page and get a developer sandbox account.

And yes the most of all Google will be open sourced and even you can develop a site like Google Wave and even can compete against Google if you have guts lolz J. And as far I know Google have allowed competition for the fist time in their field. In Short google went this to be a open source as the email protocol

httpvh://www.youtube.com/watch?v=v_UyVmITiYQ

Here is the video that I recommend you to must watch. It’s a bit long video of about 1 hr 20 min but it’s worth watching. If you really want to know how Google Wave works then do see this video. After watching this video you will be in love with the Big Google Wave.

]]> http://www.tips4web.com/google-wave-next-generation-communication/feed/ 0 http://www.tips4web.com/promote-your-website-with-link-building-2/ http://www.tips4web.com/promote-your-website-with-link-building-2/#comments Wed, 03 Jun 2009 07:48:22 +0000 Informer http://www.tips4web.com/promote-your-website-with-link-building-2/

One of the important tools of web promotion is link popularity. Link popularity refers to the popularity of your website which otherwise depends on the number of websites joining your website. To improve or to provide your website with top search engine ranking, search engine optimization is required. Link buildings play a very important role in search engine algorithm or in other words search engine ranking formulae. Link building may be defined as a process of getting links to your site from different other sites.  Link building not just improves your sites search engine ranking, but it also enhances the traffic volume to your site as well as site popularity.

Different Types of Link Building

•    RECIPROCAL LINKS: Are the sites which link to your site & in return your site links back to that site. In other words there is exchange of links. Although these links are not as effective as the back links but still they are important because they are permanent as well as cost effective.

•    INBOUND OR BACKLINKS: Are those which are one way i.e. links from other sites link to your site but in return your site does not link back. These links are any time more valuable & effective than the reciprocal links.

•    THREE WAYS LINK BUILDING: Is a tedious and a time consuming process with three domain names which would get one way link with each other.

Link Building Strategies / Techniques

•    Link building directories: By submitting in free link directories which have high page rank & high quality you could start the process of link building.

•    Outsourcing: Outsourcing your link with trustworthy services & individuals might be an expensive process but this would provide you with quality links instead of receiving low quality links at low price, which might otherwise harm your website.

•    Article writing: It is one of the most effective methods of link building. A relevant and an informative article is what visitors actually look for .This increases your sites popularity. The content rich in keywords have high keyword density, another factor required for link building.

•    Link building is a very crucial part of search engine optimization. It is not an easy task & requires a c-class IP address and an appropriate anchor text for submission to high quality directories. Apart from the quantity, quality factor needs to be kept in mind. It is an art and requires the assistance of a professional SEO.

The need to establish standards

The use of search engine spiders crawl the Internet, positioning them Indexed web pages to the database.  This process needs a lot of resources, and sometimes, you want to index the page is ignored, without being included with the page.  Search for changes in the network with the new web page, record them and the corresponding Google Googlebot spider classification is a typical example.

Site map for the site which clearly should be the index page, as well as the site has been added which provides a new method.  Basically, its search engine and Web site to provide a communication channel.  In theory, it is the search engine spiders by reducing the processing resources to alleviate the burden of the object, but can not replace the current site map crawling process.

About Site Map

Site Map is an XML document which contains a web site URL and the associated attribute table, the index should be a detailed description of what a particular site.  Site Map must be UTF-8 encoding.  The following is a site map XML file needed elements:

• <Urlset> – site map file to the start and end tags, start tags must include the namespace (xmlns) attribute.
• <Url> – the document that contains every page of this element.
• <Loc> – a document specified in the actual address of the page.  It is a subset of elements <url> elements.

Document the following optional elements:

• <Lastmod> – <url> elements of a subset of elements.  It was last updated the page specified time.
• <Changefreq> – <url> elements of a subset of elements.  It has been designated by the frequency of page updates (regular, hourly, daily, weekly, monthly, and has never been updated annually).
• <Priority> – <url> elements of a subset of elements.  Its designated website pages in relation to the importance of other pages, valid values for the 0.0-1.0, the default value of 0.5.

Samples of the following Web site explains how to map a sample page in the application of these elements.  It assumes that the Web site as a designated home page, as well as its update frequency, last update time and the priority sites.

<? xml version = “1.0″ encoding = “UTF-8″?>

<urlsetxmlns=”http://www.sitemaps.org/schemas/sitemap/0.9″>

<url>

<loc> http://www.test.com/ </ loc>

<lastmod> 2006-11-20 </ lastmod>

<changefreq> daily </ changefreq>

<priority> 0.3 </ priority>

</ url>

</ urlset>

Sitemap file from your own decision, but it determines the location of the document may contain the URL set.  For example, if the above sample file is located in Site Map http://www.test.com/sitemap.xml, then the site map file may contain any http://www.test.com/ at the beginning of the URL.  Therefore, we propose that the site map file stored in the root site.  Site Map file size should not exceed 10MB.  If a file beyond this limit, we need to use gzip to compress.

The establishment of a Site Map

Since the site map in XML-based, you can easily use any text editor to create and edit them, but also specialized tools can be applied.  The following list provides some of the current tools:

• Map node: one to generate and verify XML documents site map tool
• Gsitemap: an application. NET Framework to establish the site map generation tool.
• GSiteCrawler: a site map file to generate the Windows tool.
• phpSitemapNG: PHP with the preparation of a free site map generator.
• Google Sitemap Generator: A site map can be used to generate the Python script file.

Circular search engine

The establishment of a site map file, it should be submitted to a search engine.  Each search has its own interface to submit the site map.  Google’s Webmaster Tools to concentrate contains a site map submitted to the page.  Before the use of an account you must be registered.  Yahoo also has a free site map to submit the page, but the same must be registered before an account.  Will follow the example of other search engines Google, Yahoo and MSN, to provide similar functions.

Other tools

Indexed pages search engine to crawl the process of slow and needs a lot of resources.  Site map for the site specified what should be included which provides a way to search.  They set up the same as a simple XML text files, but there are many tools can help you create a site map file.  Now, they only deal with the current process to add.

ShareThis

Source
Establish of Sitemap

There are some advanced things you can do when considering your link text that will allow you to gain even more benefit from them. Lets look at some ideas and suggestions, as well as think about some more high levels strategies revolving about link text.

Place Plain Text Around Links

Placing some plain text next to your links is better than just having bare links all over the place. Many search engines use the text surrounding a link to help figure out what a link is about. Although I would not go so far as to place plain text in your navigation menu, and it may not look good down near the bottom of your page near the copyright notice, but when available, its always a good idea to add a line of descriptive text next to your link.

Varying Your Destination URL?

There has been alot of debate in the last few years about whether when you are out there submitting your links, asking for links, and linking your own pages up, whether you should change out the desination URL. What I mean is, ‘domain.com’ and ‘www.domain.com’ usually point to the exact same place. Also depending on my website’s setup, something like ‘domain.com/index.php’ or ‘domain.com/index.htm’ also points to the same place.

So should you be changing up the destination URL constantly? No. Should you change it up a bit here and there? Yes. You should find the scheme that you prefer most, and stick with that 80-90% of the time. So if you decide that you like the ‘domain.com’ version better than the ‘www.domain.com’ version, then stick with that most of the time. Sticking with the same scheme will give you better search engine results in general, however you should vary it using the different flavors of your URL at least 10% or so of the time.

How Many Links Per Page?

This is certainly a good question, and something that really deserves its own page. However with the present state of the web, I would personally not put more than 30 links on one page at any given time. Now there may be some exceptions to this. For instance your own sitemap page, or a page that has 50 links that all point to other areas of your site. Thats not a big deal in the eyes of the search engines. What I would avoid however, is placing large amounts of links to external sites on one big page. You don’t want to have your page be viewed as some sort of link farm right?

What If My Link Has To Be My Name?

Blog comments are a common place where there are links that are expected to be a name. Many blog systems allow you attach a link to the name you use to post comments. This free link of sorts tends to be highly abused however, and the blog owners do not look too kindly upon people who jam a bunch of keywords in as their ‘name’ when posting comments. This smells like spam and they will delete and/or blacklist you in a second. However, you can often give yourself a ‘title’ type of name. As long as you are backing it up with a long,useful,relevant comment that blows away all the other comments on the page, most blog owners would be happy to put up a comment like that.

Examples of good names to use would be ‘Sewing Sally’ instead of just ‘Sally’. Yes you are still calling yourself Sally, but at least you have the keyword ’sewing’ now added to the link as well. Other examples that you can try would be things like ‘Cooking Fan’, ‘Joe the Plumber’, or ‘Marketing Guru’.

I have been using dawjee.com to analyse search terms for a while now – toying with keyword suggestions and looking for interesting patterns or anomalies in the result data (e.g. pages with query string data often report an erroneous Google Page Rank of zero).

After discussing some findings with a colleague, I realised a lot of what I’m learning will also be of use and/or interest to other webmasters. I shall therefore catalogue all future findings in a series of articles / blog posts.

This first article shall observe the current Google results for the term “dandruff”.

At the of writing, the top 3 results are:

Position 1

Title : Dandruff – Wikipedia, the free encyclopedia

URL : http : // en . wikipedia.org/wiki/Dandruff

PR 6

Position 2

Title: Dandruff – MayoClinic.com

URL: http : // www . mayoclinic.com/health/dandruff/DS00456

PR 4

Position 3

Title: Dandruff

URL: http : // www . coolnurse.com/dandruff.htm

PR 5

(PR = Google Page Rank)

What’s interesting is that P2 (Position 2) has a lower PR (Google Page Rank) than P3 (Position 3). Both web pages contain Dandruff in their TITLE tags and urls. So a cursory glance would suggest that P3 should have a better (i.e. lower) rank than P2.

From taking a look at both pages, both are on topic and contain genuine information about dandruff. P2 is only a small page, and is part of a full article (i.e. many small interlinked dandruff pages). While P3 is a full article on a single page.

The keyword densities are 4.86% for P2 and 6.39% for P3. Neither page makes obvious use of contrived keyword stuffing.

However, if we drill down into the data for each result (use the small magnifying glass next to each result, or follow the link the Resources list below), we’ll find some significant differences in their hosts and backlinks.

P2 has 6 Google backlinks, with 4,990 to its host. While P3 has 3 and 270 respectively.

P2 also dwarfs P3 in terms of the number of pages Google has indexed (Pages in Host), and also has a higher host page rank 7 (vs 5 for P3).

To summarise, P2 may have a lower page rank, but it has a few more links and its home page is overwhelmingly more popular than P3’s. It is also part of a set of pages about dandruff. Instead of having the full article on a single page, it has been split among many. P2 therefore has several interlinked pages about dandruff.

It therefore appears that the pages hosted on popular (in terms of search engines) web sites can rank better than other pages, even if they have lower Google Page ranks. Splitting up articles into many small pages helps as well.

Have you ever tried to exchange links with other websites? The process usually requires finding a website that has a link directory and then contacting the webmaster by email to request a link exchange. If you are lucky, they may respond to your email within a week or two depending on how busy they are. If they do reply with their link information, you will then need to manually add them to your link directory page. Then there is the process of checking for dead links on a weekly basis to make sure you are not giving away free links. ExpressLinkExchange does all of this for you.

Their programmers developed ExpressLinkExchange to be a fully automated system that allows you to get new links to your website while you are working, playing golf, out to lunch, even while you are sleeping! Once you subscribe to their service, within 15 minutes you can have up to 7,841 of their members linking to your website. Setup is made easy by using their Easy Setup Wizard.

All you have to do is enter your website address, website title, a small description of your website, and choose a directory category for your website to display in. Then you have the option of using their code to create a links page on your website, or allowing them to host a links page for you on their free web hosting service. Finally, you add a link to your new links page to your home page, and you are done!

If you prefer to exchange links exclusively with websites that are relevant to your website theme, you can use their Exclusion Filters™ to deny exchanges with websites and even entire categories that you do not wish to exchange links with. Some members use the filters to gradually increase their incoming links by releasing one category each week or so. You can also choose to either automatically accept link exchanges with all new members as they subscribe or make them wait until you approve each new member. All of this is done from within your member control panel.

Because to get started, getting 1000’s of inbound links instantly on your website, Visit: http://www.ExpressLinkExchange.com now!

Shanon Sandquist is CEO of Homeworkers Publications. A company that has been in business for six years.

You can on your site to create a plain text file robots.txt, statements in this document in the site visit do not want to be part robot, so that the site of some or all of the content on the search engines do not have to be included, or designated search engine only the contents of the specified record.  robots.txt file should be placed on the root site.

When a search robot (some call search spiders) to visit a site, it will first check the site under the root directory of the existence of robots.txt, if it exists, the search robot will be in accordance with the contents of the document to determine the visit scope; If the file does not exist, then the search robots to crawl along the link.

robots.txt file:

“Robots.txt” file that contains one or more of the records through the blank lines to separate (by CR, CR / NL, or NL as at the end), each record format is as follows:

“<field>: <optionalspace> <value> <optionalspace>”.

In that paper, you can use # for comments, the specific methods and the use of UNIX in the same practice.  The document is usually recorded in one or more lines of User-agent year, followed by a number of Disallow lines, as follows:

User-agent:

The value of the search engine robot is used to describe the name of the “robots.txt” file, if the number of User-agent records that have more than one robot will be subject to the restrictions of the agreement, the document, at least There is a User-agent record.  If the value is set to *, the agreements are valid for any robot in the “robots.txt” file, “User-agent: *” This can only have a record.

Disallow:

The values do not wish to be used to describe a visit to the URL, the URL can be a complete path, it could be a part of, any Disallow the URL at the beginning will not be access to the robot.  For example, “Disallow: / help” on / help.html and / help / index.html search engines are not allowed to visit, and the “Disallow: / help /” allows robot to visit / help.html, and not be able to access / help / index . html.  Disallow any record is empty, that all parts of the site are allowed to be visited, in the “/ robots.txt” file, at least record a Disallow.  If the “/ robots.txt” is an empty file, then for all the search engine robot, the site is open.

For example the use of robots.txt file:

Example 1. A ban on all search engines to visit any part of the site to download the robots.txt file User-agent: * Disallow: /

Example 2. To allow the robot to visit all (or also can be used to build an empty file “/ robots.txt” file) User-agent: * Disallow:

Example 3. To prohibit access to a search engine User-agent: BadBotDisallow: /

Example 4. Permit to visit a search engine User-agent: baiduspiderDisallow: User-agent: * Disallow: /

Example 5. A simple example in this case, there are three directory of the Web site search engine to limit access so that search engines will not visit the three directories.  It should be noted that a directory for each statement should be kept separate, and not written in “Disallow: / cgi-bin / / tmp /”.  User-agent: * after a special meaning, representing “any robot”, so the document can not be “Disallow: / tmp / *” or “Disallow: *. gif” it was recorded there.  User-agent: * Disallow: / cgi-bin/Disallow: / tmp / Disallow: / ~ joe /

Robot special parameters:

1. Google

Allow Googlebot:

If you want to block in addition to all the roaming outside Googlebot to access your pages, you can use the following syntax:

User-agent: Disallow: /

User-agent: Googlebot

Disallow:

Googlebot follows the line of its own point, rather than point to the line of all robots.

“Allow” extension:

Googlebot identifiable as “Allow” extension of the robots.txt standard.  Other search engine bots may not be able to identify this extension, so please use your interesting to find other search engines.  ”Allow” the role of line with the principle of “Disallow” line, like.  You want to allow only listed in the directory or page you.

You can also use “Disallow” and the “Allow”.  For example, to intercept a subdirectory in the page other than all the pages, you can use the following entries:

User-Agent: Googlebot

Disallow: / folder1 /

Allow: / folder1/myfile.html

These entries will be in addition to intercept folder1 directory of all the pages outside myfile.html.

If you want to block Google’s Googlebot and allow the other robots (such as Googlebot-Mobile), can use the “Allow” rules to allow access to the robots.  For example:

User-agent: Googlebot

Disallow: /

User-agent: Googlebot-Mobile

Allow:

Use * to match its character sequence:

You can use an asterisk (*) to match the character sequence.  For example, to block all private visit at the beginning of the subdirectory, use the following entries:

User-Agent: Googlebot

Disallow: / private * /

To block all contain a question mark (?) Visit the web site, you can use the following entries:

User-agent: *

Disallow: / *?  *

Using the $ character matches the end of the URL

You can use the $ character the end of the URL specified with the matching characters.  For example, to block to. Asp at the end of the URL, you can use the following entries:

User-Agent: Googlebot

Disallow: / *. asp $

You can match this model used in conjunction with the Allow directive.  For example, if?  That a session ID, you can exclude all of the URL contains the ID to ensure that Googlebot will not crawl duplicate pages.  However, in order to?  At the end of the URL may be that you want to include the version of the page.  In this case, the robots.txt file can be set as follows:

User-agent: *

Allow: / *?  $

Disallow: / *?

Disallow: / *?  And his party will block contains?  Website (specifically, it will block all your domain name at the beginning, followed by any string, followed by a question mark (?), And then the string is arbitrary URL).

Allow: / *?  $ And his party will be allowed to contain any?  At the end of the web site (specifically, it would allow to include all your domain name at the beginning, followed by any string, followed by a question mark (?), There is no question mark after the character of the site).

Sitemap Site Map:

Site Map for the support of the new approach is the robots.txt file, including direct links sitemap file.

Like this:

Sitemap: http://www.eastsem.com/sitemap.xml

Expressed support for the current search engine company Google, Yahoo, Ask and MSN.

However, I would suggest or submit to Google Sitemap, which features a lot of links you can analyze the state of

Robots.txt benefits:

1. Almost all the search engines gives Spider follow robots.txt crawl rules, search engine Spider agreement to enter a Web site that is the entrance to the site’s robots.txt, of course, the prerequisite is the existence of the website this document.  Robots.txt is not configured for the site, Spider will be redirected to a 404 error page, the relevant studies have shown that if the site uses a custom 404 error page, then the Spider will be regarded as its robots.txt– although the is not a pure text file – Spider Index This site will bring big problems, the impact of search engine included on the site page.

2. Robots.txt to stop the unnecessary occupation of the search engines valuable server bandwidth, such as email retrievers, the majority of this type of search engine sites is meaningless; Another example image strippers, for most types of non-graphics Web site for its and has little significance, but a considerable amount of bandwidth.

3. Robots.txt to stop search engine to non-public page crawling and indexing, such as the site background processes, management procedures, in fact, for some in the operation of the site have a temporary page, if not configured robots.txt , search engines and even those temporary files will be indexed.

4. For the rich, there are many pages of web sites, configure the robots.txt is more important significance, because very often a search engine of its Spider face tremendous pressure to give Web site: Spider-like visit to the flood, if not checked and even affect the normal web site visit.

5. Similarly, if the existence of duplicate content sites, use the robots.txt page limit will not be part of search engine indexing and recorded, can be avoided by the search engine site duplicate content on penalties to ensure that Web site’s ranking will not be affected.

the risks associated with robots.txt and solutions:

1.?????everything, robots.txt at the same time also brought a certain degree of risk: the attacker also pointed out the site’s directory structure and location of private data.  Although the Web server’s security configuration properly under the premise of this is not a serious problem, but those ill reduced the difficulty of the attack.

For example, if the site privacy data www.yourdomain.com / private / index.html visit, then the settings in the robots.txt may be as follows:

User-agent: *

Disallow: / private /

In this way, an attacker can simply look at robots.txt to know the content you want to hide where the input in the browser will be able to visit our www.yourdomain.com/private/ did not like the content.  Of this situation, the general approach taken is as follows:

Set access permissions on the / private / content in password-protected so that attackers will not be able to enter.

Another approach is to the default directory changed its name to the main document index.html other, for example, abc-protect.html, so that the content will become the address www.yourdomain.com / private / abc-protect.htm, At the same time, the production of a new index.html file, the content along the lines of “you do not have permission to access this page” like, so that an attacker because I do not know the actual file name and do not have access to private content.

2. If the settings wrong, will lead the search engine will index all the data deleted.

User-agent: *

Disallow: /

The above code will be banned from all of the search engine index data.

Currently, the vast majority of search engine robots have to comply with the rules of robots.txt, and the Robots META tags are not currently supported, but is gradually increased, such as the well-known search engine on the full support of GOOGLE and GOOGLE also adds a command “archive”, can be restricted to whether or not to retain GOOGLE snapshot page.  For example:

<META NAME=”googlebot” CONTENT=”index,follow,noarchive”>

That crawl the site page and link pages to crawl along, but not to keep GOOLGE web page snapshot of the page

ShareThis

Source
How to use Robot.txt