MissingSchema: Invalid URL #174
Description
Hi Kyle,
Was just doing a new Docker install and start to get some errors below.
The only change from the Dockerfile was to add a root password and remove ENTRYPOINT and CMD.
Dockerfile changes.
RUN echo "root:Docker!" | chpasswd
ENTRYPOINT ["maltrieve"]
CMD ["-d", "/archive/samples", "-l", "/archive/maltrieve.log"]
./maltrieve.py -x
Processing source URLs
Completed source processing
/usr/local/lib/python2.7/dist-packages/bs4/init.py:166: UserWarning: No parser was explicitly specified, so I'm using the best available HTML parser for this system ("html5lib"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently.
To get rid of this warning, change this:
BeautifulSoup([your markup])
to this:
BeautifulSoup([your markup], "html5lib")
markup_type=markup_type))
Downloading samples, check log for details
Traceback (most recent call last):
File "./maltrieve.py", line 514, in
main()
File "./maltrieve.py", line 503, in main
if save_malware(each, cfg):
File "./maltrieve.py", line 323, in save_malware
stored = upload_vxcage(response, md5, cfg) or stored
File "./maltrieve.py", line 253, in upload_vxcage
response = requests.post(url, headers=headers, files=files, data=tags)
File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 109, in post
return request('post', url, data=data, json=json, *_kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 50, in request
response = session.request(method=method, url=url, *_kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 451, in request
prep = self.prepare_request(req)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 382, in prepare_request
hooks=merge_hooks(request.hooks, self.hooks),
File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 304, in prepare
self.prepare_url(url, params)
File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 362, in prepare_url
to_native_string(url, 'utf8')))
requests.exceptions.MissingSchema: Invalid URL 'True/malware/add': No schema supplied. Perhaps you meant http://True/malware/add?
$ cat maltrieve.cfg
[Maltrieve]
dumpdir = archive
logfile = maltrieve.log
logheaders = true
User-Agent = Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0)
viper = http://172.16.237.160:80
cuckoo = http://127.0.0.1:8090
vxcage = http://172.16.237.159:80
crits = https://127.0.0.1
crits_user = maltrieve
crits_key = <api_key>
crits_source = maltrieve
Filter Lists are based on mime type NO SPACE BETWEEN ,
black_list = text/html,text/plain
white_list = application/pdf,application/x-dosexec,application/x-rar-compressed,application/octet-stream,application/zip,application/x-zip-compressed,application/x-zip,application/msword,application/msexcel,application/msaccess,application/java-archive,application/postscript,application/rtf,application/x-macbinary,application/x-sh
I was also get this error
Downloading samples, check log for details
/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
needed to install
apt-get install libffi-dev openssl-dev
pip install requests[security]
Thanks in advance,
Jon