1.6.0:

* REST API endpoint is available via /api/v1/
  Eg. https://archive.yourdomain.com/api/v1/get_domains


1.5.0:

* Install the python3-mysqldb package (Ubuntu)

  On centos7 and 8 run the following if you don't have this pip installed:
  pip3 install https://download.mailpiler.com/generic-local/mysqlclient-1.4.6-cp36-cp36m-linux_x86_64.whl

* Add the following entries to the cronjob of piler user on the worker nodes:

  */10 * * * * /usr/bin/find /var/piler/error -type f | wc -l > /var/piler/stat/errors
  2,22,42 * * * * /usr/libexec/piler/import.sh

* Introduced smtp acl list, and obsoleted tcp_wrappers

* Parse Sender: header and give it preference over From: header

* pilerimport puts legacy emails to a top level dir based on their Date: header

* pilerexport may digitally sign the exported zip file

  openssl genpkey -out /etc/piler/signkey.priv -algorithm rsa -pkeyopt rsa_keygen_bits:4096
  openssl rsa -in /etc/piler/signkey.priv -pubout -out /etc/piler/signkey.pub
  chown piler:piler /etc/piler/signkey.*

  sign: pilerexport -W customer1 -f some@user -z emails.zip -p /etc/piler/signkey.priv
  verify: openssl dgst -sha256 -verify /etc/piler/signkey.pub -signature emails.zip.sig emails.zip

  See the following for more:
    https://www.zimuel.it/blog/sign-and-verify-a-file-using-openssl
    https://opensource.com/article/19/6/cryptography-basics-openssl-part-2

* pilerexport may export EML files to a zip file. Use -z <zipfile>, eg.

  pilerexport -W customer1 -f some@user -z first100.zip

  Note that piler user must be able to create the zip file at the
  given path.

* Added RFC3161 timestamp support

  Run the following sql query on all piler related databases:

  alter table customer_settings add column `timestamp` varchar(6) default null;

* Preview body of email by hovering mouse over subject lines of the search results

* Added support for custom date format and timezone for users

  Run the following sql query on all piler related databases:

  alter table user_settings add column date_format char(12) default 'Y.m.d';
  alter table user_settings add column timezone varchar(32) default null;

* Last search is saved in a cookie, and restored when the user logs in again.

* Exclude any email address longer than 41 characters.
  SphinxQL breaks the query in case of any email 42+ characters long.

* Added support for zipped store directories.

  You may consolidate the zillions of .m files in /var/piler/store/<customerid>/
  to a series of zip files, eg.

  /var/piler/store/somecustomer/00/5f3/00 -> /var/piler/store/somecustomer/00/5f3/5f3_00.zip
  /var/piler/store/somecustomer/00/5f3/01 -> /var/piler/store/somecustomer/00/5f3/5f3_01.zip
  ...

  To enable the feature set consolidated_store=1 in /etc/piler/piler.conf
  Then run /usr/libexec/piler/compact-store-dir.sh 5f3 "somecustomer"

  Note that you may compact any top level dir except the last one which
  is currently used.

* Improved the encryption resilience by prepending the data to be encrypted with a block of garbage

* Switched from Blowfish encryption to AES-256. It's backward compatible.

  Fix the attachment table in all piler related databases:

  alter table attachment add column `e` char(1) default 'b';

* IMAP import over the gui

  To enable the feature for admin users set the following in /etc/piler/config-site.php:

  $config['ENABLE_IMPORT'] = 1;

  Also fix the import table in all piler related databases:

  alter table import add column `customer` varchar(255) default null;

  If you have a multinodes layout, then run the following command on the
  worker node which you want to run import jobs on (otherwise it won't
  process the import jobs):

  touch /etc/piler/IMPORT_HOST

* Discard short header only emails not having a Message-ID: line to prevent them
  filling up the error directory.


1.4.9:

* New customer form supports specifying the admin@local password
  for the customer at the 'Admin user password' input field.

* Support for sphinx-3.3.1

  If you upgrade piler, then be sure to set the following in /etc/piler/config-site.php
  to keep using the legacy full text search columns in sphinx:

  $config['SPHINX_STRICT_SCHEMA'] = 0;

  Note that enabling SPHINX_STRICT_SCHEMA (by setting it to 1) breaks the sphinx
  index data, and they must be recreated with reindex which takes some time.

* Added support for multiple master nodes

  Add the following line to /etc/piler/config-site.php on the current master/gui node
  if you have a multinode layout:

  $config['PRIMARY_MASTER'] = 1;


  To setup an additional, secondary master node set the following to config-site.php:

  $config['NODE_TYPE'] = MASTER;
  $config['PRIMARY_MASTER'] = 0;
  $config['DB_HOSTNAME'] = '<primary master>';
  $config['MASTER_NODE'] = '<primary master>';
  $memcached_server = ['<primary master>', 11211];

  In the above example the secondary master will use the mysql and memcached servers
  running on the primary master node.

* Fixed space projection calculation

* Added zipkin compatible tracing support for the GUI

* Don't create a default auditor@local account when creating a new tenant/customer

* Fixed unique constraint on accounting table:

-  UNIQUE (`date`, `email`)
+  UNIQUE (`date`, `email`, `server_id`)

* Fixed open_database() call to add mysqlhost parameter

* Fixed a typo in r.php

* Online users are written to and read from memcached exclusively

* Disable prometheus metrics by default. To enable set the following
  in /etc/piler/config-site.php:

  $config['ENABLE_PROMETHEUS_METRICS'] = 1;

  Also it's recommended to set some sort of access control to nginx config, eg.

        location = /metrics.php {
            # Add the IP-address of your Prometheus server
            allow 10.1.1.2;
            deny all;

            fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
            fastcgi_param PATH_INFO $fastcgi_path_info;

            fastcgi_split_path_info ^(.+?\.php)(/.*)$;
            if (!-f $document_root$fastcgi_script_name) {
               return 404;
            }

            fastcgi_pass unix:PHP_FPM_SOCKET;
            fastcgi_index index.php;
            include fastcgi_params;
        }


1.4.8:

* Fixed a bug causing bogus customer info the smtp session

* Added piler-s3 systemctl service

* Updated cdn links in layout templates, and the matching nginx config

  Be sure to update the Content-Security-Policy definition:

  add_header Content-Security-Policy "default-src 'self' 'unsafe-inline' ; object-src 'none'; base-uri 'none'; script-src 'self' 'unsafe-inline' 'unsafe-eval' code.jquery.com stackpath.bootstrapcdn.com; style-src 'self' 'unsafe-inline' stackpath.bootstrapcdn.com; font-src 'self' stackpath.bootstrapcdn.com; ";

* Introduced an improved mobile search template
  Set $config['ENABLE_MOBILE_PREVIEW'] = 1; in
  config-site.php to enable it.

* Fixed a bug in the health-check.sh script in a multinodes layout.
  It mistakenly reported a worker -> master probe result

* Introduced the Data Officer role

  You need to create the following table in all piler related databases:

  create table if not exists `deleted` (
    `id` bigint unsigned not null unique,
    `requestor` varchar(128) not null,
    `reason1` varchar(255) not null,
    `date1` int unsigned default 0,
    `approver` varchar(255) default null,
    `reason2` varchar(255) default null,
    `date2` int unsigned default 0,
    `deleted` tinyint(1) default -1,
    key (`id`),
    key (deleted)
  ) Engine=InnoDB;

  Add the following rewrite to the webserver config /etc/piler/piler-nginx.conf,
  then reload the webserver:

  rewrite /rejectremove.php /index.php?route=message/rejectremove;

* Using bootstrap 3.4.1 minified js from CDN

1.4.7:

* Fixed customer id check

* Fixed creating local users in multitenancy mode

* Added backup codes for Google Authenticator

  You need to create the following table in all piler related databases:

  create table if not exists `ga_backup_code` (
    `username` varchar(64) not null,
    `code` char(6) not null,
    `used` tinyint default 0
  ) Engine=InnoDB;

  create index ga_backup_code_idx1 on ga_backup_code(username);

* Added systemd service files to /usr/libexec/piler

* S3 stuff uses /var/piler/s3 directory

  mkdir /var/piler/s3
  chown piler:piler /var/piler/s3
  chmod 700 /var/piler/s3

* S3 upload script (s3up.py) upgraded to use python3.

  You need the minio python module to use it, eg. pip3 install minio

* imapfetch.py and pilerpurge.py uses /usr/bin/python3 as well

* Introduced the search history feature

  Set ENABLE_MEMCACHED in either /etc/piler/config-site.php or in the per customer settings, and be sure to install
  the php-memcached package. It puts the search terms to memcached to offers them in reverse order, latest on top.

* Fixed Google authenticator for ldap logins

* Introduced Auth0 support for Azure
  Set ENABLE_AUTH0 and the appropriate AUTH0 variables in either /etc/piler/config-site.php or in the per customer settings

* Default Sphinx version is set to 311 in /var/piler/www/config.php
  If you have 2.2.x version of sphinx, be sure to add the following line to /etc/piler/config-site.php:

  $config['SPHINX_VERSION'] = 220;

* Added tika and disk info to node health status output

* Introduced the default SPHINX_WORKER_LISTEN_ADDRESS variable to set the sphinx listen address on the worker nodes

  $config['SPHINX_WORKER_LISTEN_ADDRESS'] = '0.0.0.0';

1.4.6:

* The gui provides output for prometheus at /metrics

  Upgrade:
  - Add the following to piler's crontab on the worker nodes:
    10 * * * * /usr/bin/find /var/piler/error -type f | wc -l > /var/piler/stat/errors


1.4.5:

* Introduced new config variable USE_SMTP_GATEWAY


1.4.4:

* Fixed ldap authentication bug (1f828af)

* ldap table has new field (ldap_port) defaults to 389 (b094020)

  Run the following sql statement on all piler related databases:
     alter table ldap add column `ldap_port` int default 389;