陈鲁豫,Java爬取并下载酷狗TOP500歌曲,活死人之地

Java爬取并下载酷狗TOP500歌曲

是这样的,之前买车送的废物记载仪不能用了,这两天决然买了好点的记载仪,带导航、音乐、蓝牙、4G等功用,深思,已然有这些功用就运用起来,用4G听歌有点奢华,就预备去酷狗下点歌听,竟然都是需要办会员才干下载,并且vip一月只能下载300首,我这么穷又这么抠怎样或许冲会员,所以百度搜了下怎样免费下载,都是python爬取,尽管也会一点,可是电脑上没装置python,再装置再研讨感觉有点费力,所以就花了半小时做了这个爬虫,技能一般,只记载剖析完结进程,大牛请绕行。其间用到了一些库,包含:jsoup、HttpClient、net.sf.json咱们能够自行去下载jar包。

1、剖析是否能取得TOP500歌单

首要,翻开酷狗主页检查酷狗TOP500,说好的500首,怎样就只有22首呢?

Java爬取并下载酷狗TOP500歌曲

是真的只让看这些仍是能找到其他的呢,所以我就看了下这TOP500的链接:

https://www.kugou.com/yy/rank/home/1-8888.html?fro陈鲁豫,Java爬取并下载酷狗TOP500歌曲,活死人之地m=rank

能够看的出home后边有个1,莫非这是代表第一页的意思?所以我就把1改成2,进入,公然进入了第二页, 至此能够知道咱们能够在网页里获取这500首的歌单。

2、剖析找到真实的mp3下载地址(这个有点绕)

点一个歌曲进入播映页面,运用谷歌浏陈鲁豫,Java爬取并下载酷狗TOP500歌曲,活死人之地览器的控制台的Elements,搜一下mp3,很轻松就定位到了MP3的方位。

可是运用java拜访的时分爬取的html里却没有该mp3的文件地址,那么这肯定是在该页面的方位运用了js来加载mp3,那么改写下网页,子宫癌看网页加载了哪些东西,加载的东西有点多,侧重看一下js、php的恳求,首要是看里边有没有mp3的地址,剖析细节就不用说了。

终究我在列表的

https://wwwapi.kugou.com/yy/index.php?r=play/getdata&callback=jQuery191027067069941080546_1546235744250&hash=667939C6E784265D541DEEE65AE4F2F8&album_id=0&_=1546235744251

这个恳求里发现了mp3的完好地址:

"play_url": "http:\/\/fs.w.kugou.com\/201812311325\/dcf5b6449160903c6ee48035e11434bb\/G128\/M08\/02\/09\/IIcBAFrZqf2ANOadADn94ubOmaU995.mp3",

那这个js是怎样判别是哪首歌的呢,那么只或许是hash这个参数来决议歌曲的,然后到播映页面里找到这个hash的方位,是鄙人面的js里:

var dataFromSmarty = [{"hash":"667939C6E784265D541DEEE65AE4F2F8","timelength":"237051","audio_name":"\\u767d\\u5c0f\\u767d - \\u6700\\u7f8e\\u5a5a\\u793c","author_name":"\\u767d\\u5c0f\\u767d","song_name":"\\u6700\\u7f8e\\u5a5a\\u793c","album_id":0}],//当时页面歌曲信息
playType = "search_single";//当时播映

在去java爬取该网页,检查能否爬到这个hash,公然,爬取的html里有这段js,到现在mp3的地址也找到了,歌单也找到了,那么下一步就用程序赵郁鑫相片完结就能够了。

3、java完结爬取酷狗mp3

先看一下爬取成果:

找到了资源,程序完结就好说了,其间运用到了自己写的几个东西类,自己收拾点自己的东西类仍是有优点的,今后遇到什么问题就没必要从头写了,直接拿来用就能够了。没什么好说的了,下面直接贴出源码。

SpiderKugou.java

package com.bing.spider;
import java.io.IOException;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import com.bing.download.FileDownload;
import com.bing.html.HtmlManage;
import com.bing.http.HttpGetConnect;
import net.sf.json.JSONObject;
public class SpiderKugou {
public static String filePath = "F:/music/";
public static String mp3 = "https://wwwapi.kugou.com/yy/index.php?r=play/getdata&callback=jQuery191027067069941080546_1546235744250&"
+ "hash=HASH&album_id=0&_=TIME";

public static String LINK = "https://www.kugou.com/yy/rank/home/PAGE-8888.html?from=rank";
//"https://www.kugou.com/yy/rank/home/PAGE-23784.html?from=rank";


public static void main(String[] args) throws IOException {

for(int i = 1 ; i < 23 ; i++){
String url = LINK.replace("PAGE", i + 陈鲁豫,Java爬取并下载酷狗TOP500歌曲,活死人之地"");
getTitle(url);
//download("https://www.kugou.com/song/mfy6je5.html");
}
}

public static String getTitle(String url) throws IOException{
HttpGetConnect connect = new HttpGetConnect();
String content = connect.connect(url, "utf-8");
HtmlManage html = new HtmlManage();
Document doc = html.manage(content);
Element ele = doc.getElementsByClass("pc_temp_songlist").get(0);
Elements eles = ele.getElementsByTag("li");
for(int i = 0 ; i < eles.size() ; i++){
Element item = eles.get(i);
String title = item.attr("title").trim();
String link = item.getElementsByTag("a").first().attr("href");

download(link,title);
}
return null;
}

public static String download(String url,String name) throws IOException{
俾斯麦String hash = "";
HttpGetConnect connect = new HttpGetConnect();
String content = connect.connect(url, "utf-8");
HtmlManage html = new HtmlManage();

String regEx = "\"hash\":\"[0-9A-Z]+\"";
// 编译正则表达式
Pattern pattern = Pattern.compile(regEx);
Matcher matcher = pattern.matcher(content);
if (matcher.find()) {
hash = matcher.group();
hash = hash.replace("\"hash\":\"", "");
hash = hash.replace("\"", "");
}

String item = mp3.replace("HASH", hash);
item = item.replace("TIME", System.currentTimeMillis() + "");
System.out.println(item);
String mp = connect.connect(item, "utf-8");

mp = mp.substring(mp.indexOf("(") + 1, mp.length() - 3);

JSONObject json = JSONObject.fromObject(mp);
String playUrl = json.getJSONObject("data"低密度脂蛋白).getString("play_url");

System.out.print(p雷斯卿layUrl + " == ");
FileDownload down = new FileDownload();
down.download(playUrl, filePath + name + ".mp3");

System.out.println(name + "下载完结");
return playUrl;
}
}

HttpGetConnect.java

package com.bing.http;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.security.NoSuchAlgorithmException;
import java.security.cert.CertificateException;
import java.security.cert.X509Certificate;
import javax.net.ssl.SSLContext;
import javax.net.ssl.TrustManager;
import javax.net.ssl.X509TrustManager;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache黄片儿.http.HttpEntity;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.H陈鲁豫,Java爬取并下载酷狗TOP500歌曲,活死人之地ttpClie刺猬紫檀nt;
import org.apache.http.client.ResponseHandler;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.client我的青春期.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.conn.ClientConnectionManager;
import org.apache.http.con胡平和n.scheme.Scheme;
import org.apache.http.conn.scheme.SchemeRegistry;
import org.apache.http.conn.ssl.SSLSocketFactory;
import org.apache.http.impl.client.BasicResponseHandler;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.DefaultHttpCli冲砂暂堵剂ent;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.impl.conn.BasicHttpClientConnectionManager;
import org.apache.http.params.HttpParams;
/**
* @阐明:
* @author: gaoll
* @CreateTime:2014-11-13
* @ModifyTime:2014-11-13
*/
public class HttpGetConnect {

/**
* 获取html内容
* @param url
* @param charsetName UTF-8、GB2312
* @return
* @throws IOException
*/
public static String connect(String url,String charsetName) throws IOException{
BasicHttpClientConnectionManager connManager = new BasicHttpClientConnectionManager();

CloseableHttpClient httpclient = HttpClients.custom()
.setConnectionManager(connManager)
.build();
String content = "";
当我想你的时分
try{
HttpGet httpget = new HttpGet(url);

RequestConfig requestConfig 陈鲁豫,Java爬取并下载酷狗TOP500歌曲,活死人之地= RequestConfig.cust双子女om()
.setSocketTimeout(5000)
.setConnectTimeout(50000)
.setConnectionRequestTimeout(50000)
.build();
httpget.setConfig(requestConfig);
httpget.setHeader("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
httpget.setHeader("Accept-Encoding", "gzip,deflate,sdch");
httpget.setHeader("Accept-Language", "zh-CN,zh;q=0.8");
httpget.set陈鲁豫,Java爬取并下载酷狗TOP500歌曲,活死人之地Header("Connection", "keep-alive");
httpget.setHeader("Upgrade-Insecure-Requests", "1");
httpget.setHeader("User-A飞雪看市gent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36");
//httpget.setHeader("Hosts", "www.oschina.net");
httpget.setHeader("cache-control", "max-age=0");

CloseableHttpResponse response = httpclient.execute(httpget);

int status = response.getStatusLine().getStatusCode();
if (status >= 200 && status < 300) {

HttpEntity entity = response.getEntity();
InputStream instream = entity.getContent();
BufferedReader br = new BufferedReader(new InputStreamReader(instream,charsetName));
StringBuffer sbf = new StringBuffer();
voiceString line = null;
while ((line = br.readLine()) != null){
sbf.append(line + "\n");
}
br.close();
content = sbf.toString();
} else {
content = "";
}

}catch(Exception e){
e.printStackTrace();
}finally{
httpclient.close();
}
//log.info("content is " + content);
return content;
}
private static Log log = LogFactory.getLog(HttpGetConnect.class);
}

HtmlManage.java

package com.bing.html;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import o陈鲁豫,Java爬取并下载酷狗TOP500歌曲,活死人之地rg.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import com.bing.http.HttpGetConnect;
/**
* @阐明:
* @author: gaoll
* @CreateTime:2014-11-13
* @ModifyTime:2014-11-13
*/
public class HtmlManage {

public Document manage(String html){
Document doc = Jsoup.parse(html);
return doc;
}

public Document manageDirect(String url) throws IOException{
Document doc = Jsoup.connect( url ).get();
return doc;
}

public List manageHtmlTag(Document doc,String tag ){
List list = new ArrayList();

Elements eskrrrlements = doc.getElementsByTag(tag);
for(int i = 0; i < elements.size() ; i++){
String str = elements.get(i).html();
list.add(str);
}
return list;
}

public List manageHtmlClass(Document doc,String clas ){
List list = new ArrayList();

Elements elements = doc.getElementsByClass(clas);
for(int i = 0; i < elements.size() ; i++){
String str = elements.get(i).html();
list.add(str);
}
return list;
}

public List manageHtmlKey(Document doc,String key,String value ){
List list = new ArrayList();

Elements elements = doc.getElementsByAttributeValue(key, value);
for(int i = 0; i < elements.size() ; i++){
String str = elements.get(i).html();
list.add(str);
}
return list;
}

private static Log log = LogFactory.getLog(H绯红女巫tmlManage.class);
}

FileDownload.java

package com.bing.download;
import java.io.BufferedInputStream;
import java.io.BufferedOutputStream;
import java.io.File;
import java.io.FileOutputStream;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
/**干咳
* @阐明:
* @author: gaoll
* @CreateTime:2014-11-20
* @ModifyTime:2014-11-20
*/
public class FileDownload {

/**
* 文件下载
* @param url 链接地址
* @param path 要保存的途径及文件名
* @return
*/
public static boolean download(String url,String path){

boolean flag = false;

CloseableHttpClient httpclient = HttpClients.createDefault();
RequestConfig requestConfig = RequestConfig.custom().setSocketTimeout(2000)
.setConnectTimeout(2000).build();
HttpGet get = new HttpGet(url);
get.setConfig(requestConfig);

BufferedInputStream in = null;
BufferedOutputStream out = null;
try{
for(int i=0;i<3;i++){
CloseableHttpResponse result = httpclient.execute(get);
System.out.println(result.getS神曲tatusLine());
if(result.getStatusLin顾烟霍e().getStatusCode() == 200){
in = new BufferedInputStream(result.getEntity().getContent());
File file = new File(path);
out = new BufferedOutputStream(new FileOutputStream(file));
byte[] buffer = new byte[1024];
int len = -1;
while((len = in.read(buffer,0,1024)) > -1){
out.write(buffer,0,len);
}
flag = true;
break;
}else if(result.getStatusLine().getStatusCode() == 500){
continue ;
}
}

}catch(Exception e){
e.printStackTrace();
flag = false;
}finally{
get.releaseConnection();
try{
if(in != null){
in.close();
}
if(out != null){
out.close();
}
}catch(Exception e){
e.printStackTrace();
flag = false;
}
}
return flag;
}
private static Log log = LogFactory.getLog(FileDownload.class);
}

到这就完毕了,有或许有些代码没贴全,首要代码现已差不多,应该能够跑起来,多多指教。

(完)

来历:my.oschina.net/gllfeixiang/blog/2995570?p=1