SSR服务器搭建,VPS搭建SSR,SSR客户端配置,vultr搭建SSR,dedecms,织梦,科学上网,织梦伪静态规则,织梦全站伪静态,织梦手机版伪静态

nginx禁止垃圾蜘蛛访问

网络技术 baishitou 40℃ 0评论

nginx禁止垃圾蜘蛛访问,把下列代码放到你的nginx配置文件里面。

#禁止Scrapy等工具的抓取
if ($http_user_agent ~* (Scrapy|Curl|HttpClient)) {
     return 403;
}
#禁止指定UA及UA为空的访问
if ($http_user_agent ~ "opensiteexplorer|MauiBot|FeedDemon|SemrushBot|Indy Library|Alexa Toolbar|AskTbFXTV|AhrefsBot|CrawlDaddy|CoolpadWebkit|Java|semrushbot|alphaseobot|semrush|Feedly|UniversalFeedParser|webmeup-crawler|ApacheBench|Microsoft URL Control|Swiftbot|ZmEu|oBot|jaunty|Python-urllib|lightDeckReports Bot|YYSpider|DigExt|HttpClient|MJ12bot|heritrix|EasouSpider|Ezooms|^$" ) {
     return 403;             
}
#禁止非GET|HEAD|POST方式的抓取
if ($request_method !~ ^(GET|HEAD|POST)$) {
    return 403;
}

 

转载请注明:白石头博客 » nginx禁止垃圾蜘蛛访问

喜欢 (1)or分享 (0)

您必须 登录 才能发表评论!