1.6.2: * timestamp signing sorts by 'id' column * Various minor fixes and improvements Be sure to run /usr/share/piler/db-upgrade.sql on all piler related databases, eg. for i in /var/piler/sphinx/piler $(find /var/piler/sphinx/* -type d); do mysql --defaults-file=/etc/piler/.my.cnf $(basename $i) < /usr/share/piler/db-upgrade.sql done * Added support for spreading the archived files in several S3 buckets. To enable this feature, set s3_spread_in_buckets=1 in /etc/piler/piler.conf, and restart the piler-s3 systemd service (AKA the s3up.py script). This feature creates 256 buckets per organization / customer. The default behaviour is to keep all archived files in a single bucket for the given organization / customer. Add s3_spread_in_buckets=0 to /etc/piler/piler.conf if you already use S3 backend to keep compatibility. 1.6.1: * Added add_header X-Frame-Options SAMEORIGIN always; to nginx config * Introduced sender table Be sure to run /usr/share/piler/db-upgrade.sql on all piler related databases, eg. for i in /var/piler/sphinx/piler $(find /var/piler/sphinx/* -type d); do mysql --defaults-file=/etc/piler/.my.cnf $(basename $i) < /usr/share/piler/db-upgrade.sql done * imapfetch.py uses ssl/tls by default. Use the --no-ssl option to revert to cleartext It also uses imapclient python module, so be sure to install it: pip install imapclient 1.6.0: * REST API endpoint is available via /api/v1/ Eg. https://archive.yourdomain.com/api/v1/get_domains * Rewrite uri changes. Be sure to update the rewrite rules in /etc/piler/piler-nginx.conf according to /etc/piler/piler-nginx.conf.dist * Increased max_packet_size to 32M in /etc/piler/sphinx.conf * Introduced the bucket feature Be sure to run /usr/share/piler/db-upgrade.sql on all piler related databases, eg. for i in /var/piler/sphinx/piler $(find /var/piler/sphinx/* -type d); do mysql --defaults-file=/etc/piler/.my.cnf $(basename $i) < /usr/share/piler/db-upgrade.sql done See the video tutorial on the bucket feature at https://youtu.be/9U8C8u3r_Bw * Added beacon to indicate which main index is to be used Run the following command to create the default value: while read -r d; do echo main1 > "${d}/MAIN"; done < <(find /var/piler/sphinx/* -type d) * Removed the ldap_port column from ldap table The LDAP port should be part of the ldap hostname URI, eg. ldaps://ldap.host:1234 * If generating audit records is enabled, you may syslog the audit records to the mail log additionally by setting the following variable in config-site.php: $config['SYSLOG_AUDIT'] = 1; The idea is to allow you move audit records to ELK / EFK stack, Graylog or similar central logging system and make it searchable. Be sure to fix your syslog config to forward the mail log entries to the central logging system. * Introduced new piler.conf variable: tls_min_version It sets the minimum TLS protocol version the piler-smtp daemon supports. Possible values: - TLSv1 (not recommended) - TLSv1.1 (not recommended) - TLSv1.2 (default) - TLSv1.3 * Added native support to authenticate against Azure AD See https://mailpiler.com/using-azure-ad-single-sign-on-sso/ for more. * Added native support to authenticate against AWS Cognito See https://mailpiler.com/using-aws-cognito-single-sign-on-sso/ for more. * Reindex utility should cleanup the temp files * Improved the pilerexport zip writing performance * The syslog entry is more concise for the piler components, eg. piler/smtp, piler/store, piler/webui, etc. * If the Date: field contains some garbage return the current timestamp * Introduced the max release feature. It's relevant to no-tenant installations only. * SMTP protocol fix * Parser fixes * Emails having a NUL byte are redirected to the error directory 1.5.0: * Install the python3-mysqldb package (Ubuntu) On centos7 and 8 run the following if you don't have this pip installed: pip3 install https://download.mailpiler.com/generic-local/mysqlclient-1.4.6-cp36-cp36m-linux_x86_64.whl * Add the following entries to the cronjob of piler user on the worker nodes: */10 * * * * /usr/bin/find /var/piler/error -type f | wc -l > /var/piler/stat/errors 2,22,42 * * * * /usr/libexec/piler/import.sh * Add the following column to the customer_settings table, if it's not already there: alter table customer_settings add column if not exists `admin_password` varchar(128) default null; * Introduced smtp acl list, and obsoleted tcp_wrappers * Parse Sender: header and give it preference over From: header * pilerimport puts legacy emails to a top level dir based on their Date: header * pilerexport may digitally sign the exported zip file openssl genpkey -out /etc/piler/signkey.priv -algorithm rsa -pkeyopt rsa_keygen_bits:4096 openssl rsa -in /etc/piler/signkey.priv -pubout -out /etc/piler/signkey.pub chown piler:piler /etc/piler/signkey.* sign: pilerexport -W customer1 -f some@user -z emails.zip -p /etc/piler/signkey.priv verify: openssl dgst -sha256 -verify /etc/piler/signkey.pub -signature emails.zip.sig emails.zip See the following for more: https://www.zimuel.it/blog/sign-and-verify-a-file-using-openssl https://opensource.com/article/19/6/cryptography-basics-openssl-part-2 * pilerexport may export EML files to a zip file. Use -z , eg. pilerexport -W customer1 -f some@user -z first100.zip Note that piler user must be able to create the zip file at the given path. * Added RFC3161 timestamp support Run the following sql query on all piler related databases: alter table customer_settings add column `timestamp` varchar(6) default null; * Preview body of email by hovering mouse over subject lines of the search results * Added support for custom date format and timezone for users Run the following sql query on all piler related databases: alter table user_settings add column date_format char(12) default 'Y.m.d'; alter table user_settings add column timezone varchar(32) default null; * Last search is saved in a cookie, and restored when the user logs in again. * Exclude any email address longer than 41 characters. SphinxQL breaks the query in case of any email 42+ characters long. * Added support for zipped store directories. You may consolidate the zillions of .m files in /var/piler/store// to a series of zip files, eg. /var/piler/store/somecustomer/00/5f3/00 -> /var/piler/store/somecustomer/00/5f3/5f3_00.zip /var/piler/store/somecustomer/00/5f3/01 -> /var/piler/store/somecustomer/00/5f3/5f3_01.zip ... To enable the feature set consolidated_store=1 in /etc/piler/piler.conf Then run /usr/libexec/piler/compact-store-dir.sh 5f3 "somecustomer" Note that you may compact any top level dir except the last one which is currently used. * Improved the encryption resilience by prepending the data to be encrypted with a block of garbage * Switched from Blowfish encryption to AES-256. It's backward compatible. Fix the attachment table in all piler related databases: alter table attachment add column `e` char(1) default 'b'; * IMAP import over the gui To enable the feature for admin users set the following in /etc/piler/config-site.php: $config['ENABLE_IMPORT'] = 1; Also fix the import table in all piler related databases: alter table import add column `customer` varchar(255) default null; If you have a multinodes layout, then run the following command on the worker node which you want to run import jobs on (otherwise it won't process the import jobs): touch /etc/piler/IMPORT_HOST * Discard short header only emails not having a Message-ID: line to prevent them filling up the error directory. 1.4.9: * New customer form supports specifying the admin@local password for the customer at the 'Admin user password' input field. * Support for sphinx-3.3.1 If you upgrade piler, then be sure to set the following in /etc/piler/config-site.php to keep using the legacy full text search columns in sphinx: $config['SPHINX_STRICT_SCHEMA'] = 0; Note that enabling SPHINX_STRICT_SCHEMA (by setting it to 1) breaks the sphinx index data, and they must be recreated with reindex which takes some time. * Added support for multiple master nodes Add the following line to /etc/piler/config-site.php on the current master/gui node if you have a multinode layout: $config['PRIMARY_MASTER'] = 1; To setup an additional, secondary master node set the following to config-site.php: $config['NODE_TYPE'] = MASTER; $config['PRIMARY_MASTER'] = 0; $config['DB_HOSTNAME'] = ''; $config['MASTER_NODE'] = ''; $memcached_server = ['', 11211]; In the above example the secondary master will use the mysql and memcached servers running on the primary master node. * Fixed space projection calculation * Added zipkin compatible tracing support for the GUI * Don't create a default auditor@local account when creating a new tenant/customer * Fixed unique constraint on accounting table: - UNIQUE (`date`, `email`) + UNIQUE (`date`, `email`, `server_id`) * Fixed open_database() call to add mysqlhost parameter * Fixed a typo in r.php * Online users are written to and read from memcached exclusively * Disable prometheus metrics by default. To enable set the following in /etc/piler/config-site.php: $config['ENABLE_PROMETHEUS_METRICS'] = 1; Also it's recommended to set some sort of access control to nginx config, eg. location = /metrics.php { # Add the IP-address of your Prometheus server allow 10.1.1.2; deny all; fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; fastcgi_param PATH_INFO $fastcgi_path_info; fastcgi_split_path_info ^(.+?\.php)(/.*)$; if (!-f $document_root$fastcgi_script_name) { return 404; } fastcgi_pass unix:PHP_FPM_SOCKET; fastcgi_index index.php; include fastcgi_params; } 1.4.8: * Fixed a bug causing bogus customer info the smtp session * Added piler-s3 systemctl service * Updated cdn links in layout templates, and the matching nginx config Be sure to update the Content-Security-Policy definition: add_header Content-Security-Policy "default-src 'self' 'unsafe-inline' ; object-src 'none'; base-uri 'none'; script-src 'self' 'unsafe-inline' 'unsafe-eval' code.jquery.com stackpath.bootstrapcdn.com; style-src 'self' 'unsafe-inline' stackpath.bootstrapcdn.com; font-src 'self' stackpath.bootstrapcdn.com; "; * Introduced an improved mobile search template Set $config['ENABLE_MOBILE_PREVIEW'] = 1; in config-site.php to enable it. * Fixed a bug in the health-check.sh script in a multinodes layout. It mistakenly reported a worker -> master probe result * Introduced the Data Officer role You need to create the following table in all piler related databases: create table if not exists `deleted` ( `id` bigint unsigned not null unique, `requestor` varchar(128) not null, `reason1` varchar(255) not null, `date1` int unsigned default 0, `approver` varchar(255) default null, `reason2` varchar(255) default null, `date2` int unsigned default 0, `deleted` tinyint(1) default -1, key (`id`), key (deleted) ) Engine=InnoDB; Add the following rewrite to the webserver config /etc/piler/piler-nginx.conf, then reload the webserver: rewrite /rejectremove.php /index.php?route=message/rejectremove; * Using bootstrap 3.4.1 minified js from CDN 1.4.7: * Fixed customer id check * Fixed creating local users in multitenancy mode * Added backup codes for Google Authenticator You need to create the following table in all piler related databases: create table if not exists `ga_backup_code` ( `username` varchar(64) not null, `code` char(6) not null, `used` tinyint default 0 ) Engine=InnoDB; create index ga_backup_code_idx1 on ga_backup_code(username); * Added systemd service files to /usr/libexec/piler * S3 stuff uses /var/piler/s3 directory mkdir /var/piler/s3 chown piler:piler /var/piler/s3 chmod 700 /var/piler/s3 * S3 upload script (s3up.py) upgraded to use python3. You need the minio python module to use it, eg. pip3 install minio * imapfetch.py and pilerpurge.py uses /usr/bin/python3 as well * Introduced the search history feature Set ENABLE_MEMCACHED in either /etc/piler/config-site.php or in the per customer settings, and be sure to install the php-memcached package. It puts the search terms to memcached to offers them in reverse order, latest on top. * Fixed Google authenticator for ldap logins * Introduced Auth0 support for Azure Set ENABLE_AUTH0 and the appropriate AUTH0 variables in either /etc/piler/config-site.php or in the per customer settings * Default Sphinx version is set to 311 in /var/piler/www/config.php If you have 2.2.x version of sphinx, be sure to add the following line to /etc/piler/config-site.php: $config['SPHINX_VERSION'] = 220; * Added tika and disk info to node health status output * Introduced the default SPHINX_WORKER_LISTEN_ADDRESS variable to set the sphinx listen address on the worker nodes $config['SPHINX_WORKER_LISTEN_ADDRESS'] = '0.0.0.0'; 1.4.6: * The gui provides output for prometheus at /metrics Upgrade: - Add the following to piler's crontab on the worker nodes: 10 * * * * /usr/bin/find /var/piler/error -type f | wc -l > /var/piler/stat/errors 1.4.5: * Introduced new config variable USE_SMTP_GATEWAY 1.4.4: * Fixed ldap authentication bug (1f828af) * ldap table has new field (ldap_port) defaults to 389 (b094020) Run the following sql statement on all piler related databases: alter table ldap add column `ldap_port` int default 389;